index.html

<h1>Zero-to-Astro</h1>
<h2>by Ryan Farber 2023-05-04</h2>
<h3>Last modified: 2023-05-04</h3>
<h4>What is this?</h4>
<p>The goal of this site is to explain in a step-by-step manner how to perform astrophysics research.
While academic papers' methods sections should do this, these are written for experts within respective subfields.
This tends to enhance the "ivory tower" problem wherein the only way to become an expert in a specific topic
is to be trained by an existing topic. While this is arguably the most time-effective way to become a master,
several problems exist with the typical apprenticeship method. For one, the student is more-or-less at the
mercy of their supervisor. Moreover, top-rate researchers are certainly first-rate at doing but not always
in explaining (and unfortunately many in fact loathe teaching!?). Furthermore, this system perpetuates a
rich-become-richer model in academia: students who do not happen to attend the same institution as an
expert are often completely unable to learn a topic; and in choosing interns or students to hire, most
experts highly value previous research in their specific topic. In fact, the immediate impetus for my
creating this site is to apply to an exoplanet job posting. While there are plenty of folks at MPE, ESO, or
the surroundings I can probably talk to, I thought I'd also see how independent of a researcher I've become.
It's also an exercise for me to see how quickly I can learn something I've had minimal exposure to.
I also like teaching and would like to create tutorials for students to learn new topics themselves.</p>

<h5>This post's goals</h5>
<p>My goal is to reproduce figures from Birkby et al. 2013. This is Prof. Jayne Birkby's highest cited 
1st-authored paper (over 300 at time of writing according to Google Scholar; for reference that's a ton(!) 
suggesting this paper made quite an impact in the field). Since I typically spend rather too much time
reading papers (indirect procrastination) my specific goal is to not read the paper and just go to
the points of interest.</p>

<h6>Self-log</h6>
<ol>
  <li>First I scrolled down until I came across Figure 1. 
    <ul>
      <li>
For those with screen readers, this is typically
indicated in bold font and spelled out as "Figure 1." (without the quotes). I also think figure captions
are typically more important than the image itself but I'll try to explain verbally any important image
features as I go along.
      </li>
      <li>
The caption mentions the top panel is a CRIRES pipeline spectra. So probably I'll need to install
(or re-create) this CRIRES pipeline after I find the raw data in order to generate the spectrum.
The cpation also talks a lot about SYSREM. Note the characters are monospaced which typically suggests
this is some sort of code (such as a software package).
      </li>
    </ul>
  </li>

  <li>
Second I searched (Control-F / CMD-F) for Fig. 1 to find the first place the figure was
described in the text. 
    <ul>
      <li>
That was in Section 2.2 Basic data reduction. However, we still need the data to reduce.
      </li>
    </ul>
  </li>
  <li>Third I skimmed the text backward to the beginning of section 2.
    <ul>
      <li>
The second section is typically the 'methods' section in papers.
      </li>
      <li>
Towards the beginning of subsection 2.1 Observations I found that the target was HD 189733,
and the observations were part of an ESO programme (186.C-0289).
      </li>
      <li>
Theoretically, I could just email the first author for the data. In fact, most astrophysics
journals today require a Data Availability section in the paper (so I checked but this paper
is too old / doesn't have one). But a lot of scientists are
slow to respond to emails. However, even then it's no guarantee they'll find the data quickly.
This paper is 10 years old and I for example have my old paper data on a few harddrives. I know
where they are since I moved recently but otherwise it'd take me a while to dig them up.
Scientists are super busy people and typically don't enjoy the thankless task of sharing their
old data.
      </li>
      <li>
Anyway, the good news is this data was part of that large ESO programme I alluded to earlier.
I'm actually not sure what I googled to stumble upon it (searching for the program id didn't help
much) but I pretty quickly found archive.eso.org (googling eso data archive works! just be sure
not to make my dyslexic mistake of "eos" ;)
      </li>
    </ul>
  <li>Fourth, I found myself at http://archive.eso.org/eso/eso_archive_main.html
    <ul>
      <li>
Admittedly my instinctive reaction was "oh noooo" as you're provided with the equivalent
of a pilot's console - a huge mess of buttons, checkboxes, wrapped in visually disturbing colors.
      </li>
      <li>
But that's okay! Just refill your cup of water or take a deep breath. We're going to overcome this
sensory overload together :)
      </li>
      <li>
I slowly read the information from top-bottom until I arrived at the Target, Program, and Scheduling
Information section. There were a lot of fields I didn't know the information of but from the paper
I entered HD 189733 for the target in the green box on the left, the Night in the pink box on the right
(2011 08 01), as well as the program id (186.C-0289).
      </li>
      <li>
It took a moment to figure out what to do next -- the search button is/was at the top left-ish part of the
screen (above the Target Program and Scheduling Information section). So I hit that search button.
      </li>
      <li>
The results look promising! In the OBJECT column there's mostly HD 189733 which is the target data.
There's also a "SKY" entry which is probably for flatfield division and two "OBJECT" entries which might
be for calibration or background subtraction. Anyway, we'll figure those out later.
      </li>
<li>
It's also now evident that CRIRES is the name of the instrument doing the observing.
One thing that's a little weird is that a table at the bottom says the cumulative exposure time
was 4 hours whereas the paper says the observations were for 5 hours. It's possible the time allocation
committee gave them 5 hours but for some reason (e.g., clouds or the observers before them ran late, etc.)
they only were given 4 hours and the paper effectively has a typo. However I'm also wondering if
the observations began before midnight on August 10th or something like that. That'd be 2.5 hours
before the first data entry which seems a bit unlikely though. I also just realized SKY is unlikely
for flatfielding as it was during the middle of their observing run (so in the middle of the night)
so maybe that's for background subtraction. Again, we'll figure that out later.
</li>
<li>
Let's download this data!
</li>
<li>
Reading the fine print at the top of the page, it seems we need to fill out a registration
form which hyperlinks to a login page.
</li>
<li>
I clicked on "I would like to create an account". And filled out some information on the resulting
page. Note that for the password field they only allow a few symbols :(
They also allowed me to use "password" :(
I didn't submit with "password" as my password though don't worry!
The downside is I can't do my usual trick for generating a unique password per site
so I likely won't remember it. Oh well, forgot password is my fren :)
</li>
<li>
After filling out the form I then received an email (not immediately but within a few minutes)
with a link I clicked to activate my account.
</li>
<li>
Returning to the webpage with the form (I left it open; I had control-clicked on the registration
link previously). I clicked all the green boxes at the left-most column (annoying there's no
select-all option...) and then clicked the "Request marked datasets (new service)" button
which is/was immediately below the table on the left.
Note that the site is just http (not secure) so ideally do this on a burner device as ne'er do wells can
probably spoof/hack the site pretty easily...
</li>
<li>
Clicking that button brought me to a visually much more appealing page, which was also by default
in dark mode (it's daytime now but still appreciated). Apparently I selected 99 files which is
close to half a GB. So the download might take a while. Also it says that none of data has
processed data available. That's fine by me since I'd like to try to do that myself anyway - maximal
learning! :)
</li>
<li>
There were three download option. I clicked the "download zip file" option (the shell script probably
uses wget, which might be smarter for larger datasets. I don't know how that works under-the-hood but
it might be like rsync and be able to gracefully resume if the connection is intermittent. My connection
is okay though so I'll just do the ZIP. This site actually is secure which is much better. You should
probably check the address (since the previous page was not secure) to make sure it's archive.eso.org
and even then better to download on a burner device in case the site gets hacked and download malware.
I'm hoping hackers are astro lovers tho and will leave eso alone for now :)
</li>
<li>
I went for a little snack and it must've downloaded pretty fast! Was done about 8 minutes ago.
Your mileage may vary though; my download speed should be pretty good (for 2023) since I'm at an institute.
</li>
<li>
I then created a 'data' directory in this github repo, `mv`'d the downloaded zip file to here and unzipped
it (`unzip` command on mac osx 12.1 and I think on linux you can install with your package manager if you
don't have it already; you probably would have gunzip (GNU unzip) tho).
</li>
<li>
There are a lot of CRIRE*.fits.Z files. I'm not sure about the .Z at the end (maybe each file is compressed?)
but FITS is a typical astro image/file format. There's also a few readme*.txt files I'll look at next.
</li>
<li>
readme_archive.txt indicates which files weren't downloaded (if they're missing or top-secret apparently).
It didn't list any which is nice :)
</li>
<li>
The other readme didn't have anything apparently useful in it, just a list of files with name conversions
(but not apparently present for what I downloaded).
</li>
<li>
Now that we have the data, let's go back to the paper and see what they say about how they processed it.
</li>
<li>
Okay! Back at Section 2.2 of Birkby et al. 2013 it says they used version 2.2.1 of CRIRES `ESOREX` pipeline
with ESOREX in a monospace font. Let's google for that.
</li>
<li>
I found https://www.eso.org/sci/software/cpl/esorex.html which explains EsoRex stands for ESO Recipe Execution
Tool.
</li>
<li>
On the downloads page the oldest version I can see is for EsoRex 3.13.2.
So we'll hope nothing changed too drastically the past 10 years and just go with the latest
version I guess...
</li>
<li>
Downloaded esorex-3.13.6.tar.gz (which went super quick!). This was in the "Current CPL Release"
table. 
</li>
<li>
I then created a 'code' directory in this github repo, `mv`'d it to there and used 'tar xvzf esorex-3.13.6.tar.gz'
which unzips (gz stands for GNU zip) via the "z" option and then extracts the tar archive (sort of like unzipping
but note that tar doesn't compress as much as it just reconfigures directories/files into a single file) with
the "x" option. The "v" option stands for verbose and is why all the filenames show up as they're untarred.
The "f" option I don't remember, so we'll use 'man tar' to find out. There's a lot of information in manual pages
and this should be the first place you look to learn more about a command if you really want a deep understanding
of unix systems (or so the old grouches on stackoverflow say at least). Anyway I jumped down to -f which indicates
the option is for specifying the name (important!)
</li>
<li>
Anyway, now we probably need to set up / install esorex. That'll have to wait until tomorrow for me though!
</li>
    </ul>
  </li>
  <li>Day 2</li>
<ul>
<li>
Okay! Back to setting up / installing esorex. First looking at the README, it mentions I need
ISO-C++ and a C++ compiler (ideally a specific version of GNU gcc). Since I'm on a mac, I have
clang (apple's version) for gcc not that specific GNU version so if something goes wrong I might
look into this more, but it probably doesn't matter.
</li>
<li>
More importantly, I need CPL, which was also available on the same download page as esorex. So
I'll download + set up / install CPL first :)
</li>
<li>
I downloaded CPL but realized I really ought to check the md5 sum compared to the one they provide.
It's quite nice they do this which guarantees file integrity / that we're downloading what we
expect (of course someone malicious that can change the download *might* also be able to change
the md5sum displayed too...but we'll hope not).
</li>
<li>
Unfortunately 'md5sum -c esorex-3.13.6.tar.gz' gives md5sum command not found. So I googled 
"mac md5sum command not found" and the first page was a github issue for some random repo (openmaptiles),
but the important thing is that someone mentioned one can use 'brew install md5sha1sum' to install.
If you cannot use 'brew' (and are on mac) then you'll need to google how to set it up (need to
activate command line tools I think).
</li>
<li>
Anywho 'brew install md5sha1sum' installed md5sum successfully. However, ESO's recommended command
of 'md5sum -c filename' (with filename==esorex-3.13.6.tar.gz to start and later the cpl file downloaded)
Gave an 'error could not parse check file'. So I googled 'md5sum Could not parse check file'. The top
link was by oreilly.com (sort of strange...) and it mentions basic usage is 'md5sum myfile' so I removed
the -c option and it worked :)
</li>
<li>
Unfortunately, brew didn't install a man page for md5sum so I'm not sure what -c was for.
Anyway 'md5sum esorex-3.13.6.tar.gz' generated 04dba43ece4c3df2f8d9b9dcc0cd8e3a. That looks
awfully similar to what is listed on the website but how can we find out without making our
eyes bleed?
</li>
<li>
Python to the rescue! If you type 'python' on the command line you'll instead have a python command
prompt (previously we were in shell (more specifically zsh for me, possibly bash for you)). I then
ran "04dba43ece4c3df2f8d9b9dcc0cd8e3a" == "04dba43ece4c3df2f8d9b9dcc0cd8e3a" where the first string
I copy-pasted from the output of md5sum and the second string I copy-pasted from the ESO website.
That returned True meaning we're good.
</li>
<li>
I then did the same for the cpl file just downloaded (and then moved it to the code directory and
untarred - exercise to the reader: 1) try to check the file integrity of what you downloaded using
md5sum, 2) if all good in step 1, untar. Try to do this without looking above. Google if you need
help. If this takes longer than a few minutes though feel free to look above. If you have trouble
remembering tar arguments my favorite xkcd commiserates: https://xkcd.com/1168/
</li>
<li>
Anyway, the fine print under the CPL download on the ESO site mentions CPL requires CFITSIO 3.350
or later (and in case you were wondering, yes, installing / setting up software is still the hardest
part in astrophysics research in this year of 2023...)
</li>
<li>
Googling "CFITSIO install mac" brought up a nasa.gov link first. But in a rare act of scrolling down
(particularly since I'm typing this as I go which nicely slows things down a bit and I think improves
my attention span too) I noticed the third link is a homebrew formula! You can use macports if you
prefer (which apparently is what the NASA site talks through) but I'm used to using brew for pretty
much everything these days.
</li>
<li>
So I copy pasted 'brew install cfitsio' from the brew formula (note that copy-paste is your friend!
The best engineers/researchers copy-paste. For one thing, typing everything out enhances the likelihood
of an RSI (repetitive stress injury; I've unfortunately had many because I don't enjoy taking breaks...
right now I feel some pain beginning in my right hand due to all this typing). For another thing,
you don't get any points for memorizing things. I'd award more points for the person that knows how
to ~derive knowledge than one who memorizes the "answer" (that is, google your way to a brew formula
or valid tar command). Over time of using commands over and over you'll unfortunately memorize a few
(it took me a while for tars but I got there after a few years).
</li>
<li>
Anyway, 'brew install cfitsio' spit out a lot of stuff. You don't typically want to read the gibberish
but 'cfitsio --version' gave a 'command not found' error so this time we'll skim the brew output
and find the version installed. For me, that was found in this line (I looked from bottom to top, not
necessarily sequentiallyt tho): 🍺  /opt/homebrew/Cellar/cfitsio/4.2.0: 17 files, 3.1MB 
This is again higher than the version esorex requested. Hopefully that's fine but we'll keep in
mind there could be version compatibility issues.
</li>
<li>
Now I really will take a few minute break :)
</li>
<li>
The other benefit of breaks is for hydration! Anyway, now that we have cfitsio installed
let's return to cpl. Looking at the README it says that the required dependencies are
CFITSIO (all good), wcslib 4.24+ (?) and fftw 3.3.3+ (prob not installed).
</li>
<li>
I don't know what wcslib is. FFTW stand for fastest fourier transform in the west
(the creators have a nice sense of nameing :) and probably is a good idea to have
otherwise are analysis might be a lot slower.
</li>
<li>
Anyway, let's try to see if we have either of these already and/or install them.
Googling 'wcslib mac' brought up a brew recipe! Hurrah!! So, I copy-pasted
'brew install wcslib' which installed version 7.12_1.
</li>
<li>
Next, I Google'd (btw I philosophically prefer Brave search but unfortunately the results
are not nearly as useful as Google..) 'fftw mac brew' (since brew would be my preferred
way to install) and there is a formula! So actually installation is not nearly as painful
as 10-20 years ago :) 
</li>
<li>
The brew install for fftw took more than a few seconds (was installing lots of things)
so I took a break and came back. The bottom message said something about PATH so I took
a closer look at the output. It's talking about sphinx-doc which I don't care about
(it's for documentation which is important but not today!). I don't see any errors
scrolling up so we'll assume for now that fftw installed successfully.
</li>
<li>
Now let's return to installing CPL! I scrolled down and read a decent bit of the CPL
readme. It does a quite nice job of describing how to install the dependencies but it
assumes we built from source whereas we used brew. So I got to thinking: since cfitsio
was from nasa but was on brew, maybe cpl is on brew too? So I Google'd 'cpl mac brew'
and it showed up! Note that there could be multiple libraries called cpl (and generally
if the name is short or common) but the formula page links to eso.org.
</li>
<li>
So I copy-pasted the formul 'brew install cpl' and it installed 7.3.2. I'll probably delete
cpl from the 'code' directory but will leave it alone for now. I Google'd 'esorex mac brew'
but that didn't turn up a formula. So the question is whether we can actually install esorex
or if we'll have to install all the dependencies from source / do some configuring.
</li>
<li>
Note that the README mentions that when executing the configure script to specify where
cfitsio and cpl were installed. I tried 'which cpl' (and same for cfitsio) but my system
doesn't seem to know where they are. The README also mentions one can instead use CFITSIODIR
and CPLDIR environment variables. Unfortunately 'echo $CFITSIODIR' (and CPLDIR) return nothing.
Too bad, I was hoping brew handled those. Anyway, it mentions the INSTALL file has more specific
directions.
</li>
<li>
Actually the INSTALL file I have appears to be the template from GNU. Which is not immediately
helpful but it does have some interesting/useful information on generally the command flow.
We'll go ahead and give it a blind shot but probably installation will fail and we'll have to
figure out where brew installed cfitsio and cpl.
</li>
<li>
Running './configure' in the 'esorex-3.13.6' directory did indeed error, saying libcext headers
were not found. That's rather obtuse but probably refers to either cfitsio or cpl. So let's
figure out where those were installed.
</li>
<li>
I google'd "where does brew install packages" and clicked on the second link which is
apple.stackexchange.com. Generally stackexchanges are some of the best places to find clear
and concise answers to programming/software problems. It mentions the command 'brew info packagename'
will show the path. 'brew info cpl' brought up quite a few lines but does indeed mention the path
is /opt/homebrew/Cellar/cpl/7.3.2. So I set the CPLDIR environment variable to that with
'export CPLDIR=/opt/homebrew/Cellar/cpl/7.3.2'. Note that it was probably better to have googled
more generically 'where does brew install packages' than for CPL specifically as CPL was only installed
once in the past 30 days (by brew which means by me) 123 days this past year. Generally, not a ton
of people use highly specialized astrophysics software so there often isn't a ton of informational
content on how to install stuff. ESO seems to be pretty good and hopefully this will also help some people. 
</li>
<li>
Next, I similarly set my CFITSIODIR environment variable, which I'll leave as an exercise to the reader.
</li>
<li>
Note that the spelling of CFITSIODIR and CPLDIR must be exact otherwise the system won't find them.
</li>
<li>
This time running './configure' in 'esorex-3.13.6' worked! That is, it generated a Makefile and
didn't end with an error.
</li>
<li>
We'll next try running the Makefile. I'll try 'make -j 8' to run in parallel on 8 cores/threads.
If you're running on a shared system you might want to consult with your system admin how many cores
you can run on (but generally 8 is fine these days or if you want to play it safe just use 2 or 4).
</li>
<li>
The downside of 'make -j 8' (and generally, performing make in parallel) is that errors can
appear before the end of the file, as it depends which processor/thread encounters an error when.
Anyway, I scrolled up and only noticed warnings. Probably it would've been better to pipe the output
to a file so one could grep for errors. But we'll assume for now that it worked.
</li>
<li>
The next step typically is to run make install. That generated a lot of 'nothing to be done'
messages which makes me think the make command probably didn't work previously. But maybe everything is
fine. I tried 'make installcheck' as a test but that also generated a lot of 'nothing to be done' messages.
So I'm still not sure if esorex is installed or not.
</li>
<li>
Usually the result of installing from source is the creation of a 'bin' directory in the package containing
a binary with a name that matches e.g., esorex. Unfotunately no 'bin' directory was created. There are files
called esorex in an 'etc' as well as a 'src' directory. Are they the same? Unfortunately, using the 'diff'
command suggests they're different :(
Let's see if there's a hello world example or some quickstart in the documentation.
</li>
<li>
Back on https://www.eso.org/sci/software/cpl/esorex.html there is a "Using EsoRex" section. Probably
the first thing to do is 'esorex --help' just typing that of course gives command not found. It turns
out that the 'esorex' in the 'etc' directory is a text file so maybe the one in 'src' is the correct
binary!? From the 'esorex-3.13.6' directory, doing './src/esorex --help' spat out some stuff that looks
correct!
</li>
<li>
Okay! So possibly esorex is installed correctly. I've now skimmed the rest of the esorex.html page from
eso.org. There's no real hello world example that I see. One thing that looks promising is the "Sets of
frames" section, which mentions to check documentation for a given instrument to find a concrete example
for that instrument. I'll give that a shot tomorrow. (I spent 2 hours on this on day 1 and 90-120 minutes
today (including breaks); the goal is ~2 hours per day).
</li>
<li>
Day 3, session 1 (~30 minutes).
</li>
<li>
Okay! So now I'm going to try to find documentation for the CRIRES instrument.
</li>
<li>
I googled 'eso crires instrument documentation' and the first link was
an eso.org page 'Manuals - CRIRES' from there I clicked on the
Data Reduction Cookbook for Period 93 - which is again a bit ahead
of the time probably compared to Birkby+ 2013. I then skimmed
the table of contents and there's probably a lot of useful information
there.
</li>
<li>
Looking back at Birkby+ 2013 Sec 2.2 it doesn't really say much about
how esorex was used exactly. The first paragraph gives the overall
procedure which is nice and the second paragraph mentions that their
first step was removing bad columns / bad pixels which were identified
by eye. So I'm now thinking maybe I should just try to plot the fits
data in python and return to crires docs later.
</li>
<li>
I wasn't sure how to do that best so I googled 'plot fits file python'
which brought me to astropy docs.
</li>
<li>
I created a 'scripts' directory in here to store my python scripts
and copy-pasted quite a few lines from this astropy docs page
(if the link dies, hopefully you can still see my script...)
https://docs.astropy.org/en/stable/generated/examples/io/plot_fits-image.html
</li>
<li>
I then created a virtual environment (python3 -m venv env-Z2A) which is
nice for encapsulating installs and activated it (source env-Z2A/bin/activate)
I then upgraded pip (I always forget the
command but if you do 'pip install upgrade pip' then it'll fail and
tell you the correct command), installed matplotlib (pip install matplotlib)
and astropy (pip install [you fill this part in; feel free to google for help]).
</li>
<li>
I then modified the script to use the actual path to my data. That
complained about not being in 'main' -- but I had an error in my path,
but still got the same error after correcting. Odd but maybe it requires
the data being in the same directory as the script (lame if so but oh well)
. So I copied ('cp' command) one of the FITS files to the same directory
as my script and ran it again. This time I got an OSError that it's not
a simple FITS file. The error message mentions how to ignore so I'll try
that next. I just noticed that happens when I try to print fits.info(image_file)
so at least it loaded it successfully this time. Now I got empty
or corrupted FITS file error. Which is certainly possible.
</li>
<li>
So I used ls -l in the data dir to see what was up. The first file
which I was loading was 2MB whereas most were 4.4MB, so maybe there
was indeed some corruption. However, I tried the first one that was
4.4MB and got the same emtpy/corrupt file error. So then I tried a 
random file and it gave an error during the 'get_pkg_data_filename'
step. The error was from urllib and looked like it was trying to
download the file from astropy.com (weird). So maybe I was trying
to copy-paste the example too closely. Reading the top of the page
I found that Lia is one of the authors(!) and also that get_pkg_data_filename
is used to download the file -- which is not what I want since I already have
the data locally -- and astropy.io.fits is to load the file. That
sounds more like what I want so I'll open astropy.io.fits in a new
tab to see how that works. That brought up a janky API reference.
That's okay we'll just take a deep breath and read a tiny bit. Okay
open() sounds promising let's start there.
</li>
<li>
Using open also gave the empty/corrupt message. It could be true
or it could be that fits just isn't handling this correct. There's
still that .Z extension so I have a feeling I actually have compressed
FITS files.
</li>
<li>
I first tried 'unzip' but that didn't work. But gunzip did! Let's try
loading that in astropy again. That worked! Okay. Sadge astropy can't
autodetect it was zipped from the filename. astropy should be open-source
and github so I could try adding that feature. But let's not jump
down that rabbit hole quite yet.
</li>
<li>
Day 3: Part 2. Part 1 actually went 45 minutes. Start is 1:02pm, goal
is to go to 2pm. I'll save plot.py as plot_v1.py and do manual versioning
to make it easier for ppl to see the evolution of the file (sure they
could git diff but let's make it more obvious). That first version
just loaded the data. Now let's try plotting it.
</li>
<li>
Hmmm. I plotted the data in plot_v4 but it looks all grey. Maybe it
was a flatfield or some calibration frame though? I guess I'll plot
them all real quick and then inspect.
</li>
<li>
In v5 I discovered that using fits.open was much more useful
than trying to blindly get the extension correct - for most of
the files (I guess except the first one or few) the "primary" hdu
is not an image (I guess...) but the 4 chips are.
</li>
<li>
Hmmmm, only 3 primaries plotted and they seems at least sort of like
spectra. The individual chips are different series of grey it looks
like...
</li>
<li>
Figure 1 of Birkby+ 2013 refers to the plot as being CRIRES pipeline
spectra. So maybe I'd better head back to ESOREX...
</li>
<li>
I'm at a bit of a crossroads. I took another look at the CRIRES table of
contents and started skimming Section 8 which discusses reduction for
nodding (which Birkby+ 2013 says they did for their observations).
But it talks about ds9 and other tools I don't really want to need
to install. I'm a bit wary about sort of blindly using esorex. Since
there's not too many files in esorex's src I just scrolled through
er_help.c. That wasn't useful but suggests I could go through the src
files and either actually understand how esorex works or recreate
its functionality in my python script. One useful thing from the crires
manual is it mentions I need to grab calibration files from their website.
Maddeningly though they repeat an instruction (presumably for clarity) and
say the OPPOSITE of what they wrote above...and it's about pre/post April
2011 data. Since this is Feb 2011 data this matters for us...
</li>
<li>
The good news is the download page made it clear which data belongs
to our date range. Downloaded CR_PDL{A,B,C}_100727A_ALL.fits which
is valid 18 July 2010 to April 2011. Hmmmmmmm, but Birkby+ 2013 uses
CRIRES pipeline v2.2.1 and I just noticed the non-linearity cofficiencts
cube for pipeline v2.1.x is for the April 2011 - January 2012 data.
It's also possible Birkby used a later one... Well, I guess I'll
go with the data relevant to when the observations were taken - or at
least try to, it's possible esorex is too new and won't play with
these old separted "detlin" files as much as the "coeffs_cube"...
</li>
<li>
Okay so I put those non-linearity coefficients in a 'calibration'
directory but I'm still not sure where I grab the darks and flatfields.
Darks could be compiled on a different day but flatfields I think should
be same day (and maybe same day is best for darks too).
</li>
<li>
The calibration page (where I downloaded the non-linear coefficient data,
namely here: https://www.eso.org/observing/dfo/quality/CRIRES/pipeline/pipe_calib.html, which I found from the documentation pdf) mentions the calibration data (darks, flats etc.) are taken as part of the daily CRIRES plan and/or upon request; they're marked by the DPR TYPE keyword in the FITS headers. As I was writing this I realized that maybe I haven't downloaded the data yet since for the explorer maybe I only selected data relevant to that HD object. But first I'll inspect the FITS headers that I have.
</li>
<li>
Well, I manually inspected the headers for the first few primaries which
each had 0.5 second exposures and again a wealth of information there
but not what I'm looking for. I then used python to look specifically for
DPR in a header and there were a few. For DPR TYPE all but one were object
the other one was SKY which is not in the list. So I'm guessing
I need to go back to the data explorer and maybe look for all the things
on the night of/before/after and try to find the calibration data (darks,flats)
there to download. It's almost 2:30pm now so today I spent in total a bit
over 2 hours.
</li>
<li>
Day 4: Sunday May 7th 2023, 8:10am (started 8:07am)
First a quick recap. Yesterday I found the CRIRES documentation.
I also set up a python virtual environment and installed astropy
and other modules to the point I was able to plot the raw data.
I also took a look at one of esorex's source files (er_help.c)
</li>
<li>
Today I'm moving apartments so I'm going to try this in a series
of 20-30 minute sessions. The goal for this first session is to
find and download the darks/flats/etc from the eso data archive.
</li>
<li>
The wifi was slow to connect so I ended up initially at https://archive.eso.org/cms/data-portal.html
and I decided to open both the raw data query form AND the instrument specific
query form. On the instrument specific page I clicked on CRIRES.
This form lets one choose the specific DPR CATG and DPR TYPE including
specifically darks and flats! But for now I'll just try querying
by the target name (HD 189733) and the Program Id (186.C-0289).
  That returned 727 records - quite a few more than what I found before!
It looks like it returned multiple program IDs though with letters H-J
at the end.
  I'll just try searching by the date and maybe remove the target
name and program to try to find the darks and flats.
  Okay! Now there are a lot of observations of HD189733 still but there
are some WAVE,SKY objects/DPR TYPE with CALIB as DPR CATG! Still
no darks and flats though...Let's try the previous day?
</li>
<li>
July 31 2011 brought up flats and darks for july 31st. That's separated
by ~48 hours from the science observations though which have a datestamp
of August 2nd (UTC...) If I instead search August 2 2011 then it brings
up darks and flats about 6 hours after the last observation -- which
sounds reasonable :) Okay now I'll download all this stuff.
</li>
<li>
I started to mark them all 1-1 but felt annoyed so I went to the top and
looked a bit - and found the MarkAll button :)
It's now 8:28am. I'll shower while that's downloading.
</li>
<li>
Day 4 session 2, now 10:08am
</li>
<li>
Okay! I unpacked, cleaned up/sorted things in the new room and showered.
I have another load+ of stuff to grab from my old apartment and need
breakfast/lunch but I'm planning on putting another ~20 minutes in now.
</li>
<li>
The goal for this session will be to play around with the calibration
data. Probably start by creating a master dark. Anyway I first
unzipped and unfortunately they're all named as fits by date acquired
I guess. Actually that should be fine since the fits header will indicate
if they're darks or flats, etc. So let's create that master dark by
taking the median of all the darks. I'll do this in create_master_dark.py
</li>
<li>
Okay! After some iterating I know have master darks created for each chip.
The next step will be to create a master flat. I'll do that after I break
for lunch + move a bit more though. It's now 10:52am so I spent ~45 minutes
in this session.
</li>
<li>
Day 4 Session 3, start 1:43pm
</li>
<li>
So far I've spent 1h4m. The goal is to spend another 56 minutes
total today. So this session I'll aim to do 20-30 minutes. The
goal is to make a master flat.
</li>
<li>
I'm glad I carefully checked the filenames for the flats too instead
of just relying on the counts matching! It turns out I downloaded
some calibration files for the next day too! So I'll re-do the master
darks first to just use data marked as 2011-08-02 since that's
only 6 hours away from the observations whereas the others are ~30h.
Just one day probably won't change the detector dark current much
but let's see.
</li>
<li>
I don't think I have the master flat quite right yet. It's now
2:30pm so I spent about 45 minutes this session. I'll unpack
and maybe have better ideas a bit later.
</li>
<li>
Day 5, 9:47am
</li>
<li>
Well, I ~immediately realized I should just normalize to the maximum
value in each flat but continued to move, worked on my Superwinds project,
called family, dealt with logistics of a conference I'm co-organizing
for the end of the month and then it was midnight.
</li>
<li>
I have 4 meetings today (rather a lot) but my goal is still to put in 2
hours. Even better if I can do an extra 45 minutes to make up for yesterday
and I think day 2 I spent only 1.5 hours.
</li>
<li>
The median is still super small...so possibly there are some hot pixels
that are throwing off the normalization. Let's make a histogram of the
values for one chip for one frame to inspect what's going on.
</li>
<li>
Plotting the histogram indicated there are rather a lot of negative values,
which was made more clear by printing min,max,median,mean. Commenting out
the dark subtraction helped a bit. So I'm not totally sure if I need to
toss garbage pixels when constructing the dark vs tossing out negatives
after subtracting the dark. I guess if the value is negative that means
the thermal current is on average larger so it's pointless and could be
masked out.
</li>
<li>
Hmmmm the mean and median are still super small. I need to break to prep
for my first meeting today though. It's now 10:07am so I spent 20 minutes.
Maybe I'll try using esorex when I get back...
</li>
<li>
Day 5 session 2, 11:28am
</li>
<li>
I just realized that while I tossed out the negatives I forgot about
dealing with hot pixels. Let's see if those are more evident in the
histogram that excludes negatives.
</li>
<li>
The histogram still wasn't very clear but inspecting percentiles made
it clear: about 0.1% of the data (for this chip and this frame at least)
is MUCH larger than most pixels. I remembered that Birkby+ 2013 mentioned
handling bad pixels and indeed Section 2.2 they say the total of
bad pixels was 0.2-0.9%. Also note that the 0.1% above is about 5
standard deviations. So I'll do the normalization now by grabbing the
maximum value 5 standard deviations above the mean.
</li>
<li>
That was still rather small. Let me check what it is excluding zeros tho.
Nope not much bigger when masking zeros.
</li>
<li>
So I think v4 was in fact the most sensible. I was concerned then that
there were some values greater than zero. But I guess if these are
dawn flats then the sky gets progressively brighter so I actually
would need to scale down the later flats to get a fair median.
I just ran v4 again and indeed that seemed the best. So it's time
to proceed!
</li>
<li>
Okay so I have a master dark as well as a master flat. I also downloaded
some calibration files of DPR TYPE "MATRIX,INTERACTION,PRIM" as well as
"WAVE,LAMP" and additionally "WAVE,ABSORPTION-CELL". I'm not sure
what to do with those exactly. There's also those nonlinear coefficients
to use. It's now 12:10pm so I spent 40 minutes now and it's lunch time.
When i get back I'm going to read Birkby+ 2013 to prepare a presentation
for group meeting (it's an arxiv round). Probably the next most useful
thing would be to actually read the CRIRES user manual and/or source
code...
</li>
<li>
Day 6.
</li>
<li>
I had a rather longer workout than planned and also had to grab groceries
so I didn't end up doing the last 15 minutes yesterday. Anyway, I've
planned to do 3 hours today so that should get me back to an average
of 2 hours per day (and be a bit higher than that actually). I started
at 9am and did a 25 minute session, reading the CRIRES user manual.
Nice review on AO and possibly useful reading but nothing noteworthy.
Second session 9:30am through now (9:40am) and I've just read the section
on correcting the detector for non-linearity: this is supposed to be
applied to the science AND flat fields. Also note the detector glow
from dark exposures looks quite bad at the corners. It says
nodding will remove it though so I guess we'll be okay.
</li>
<li>
It's now session 3 (10a - 10:45a; now 10:02am) and I've found the
static calibration tables as a result of links in the user manual
and further link clicking around the ESO CRIRES/pipeline page.
I've learned that EsoReflex is not supported for CRIRES and that
I should be able to reduce data with EsoRex (command-line) or
Gasgano (Java based GUI tool) just as well. Anyway the calibration
table webpage is here: https://www.eso.org/observing/dfo/quality/CRIRES/pipeline/pipe_calib.html
and you have to scroll down a bit.
</li>
<li>
One other thing I learned is that data packages were provided for
data pre October 2011 such as in this work! So maybe I can find
the data packages, and otherwise I just have to do extra work
(which I've been doing).
</li>
<li>
Note from CRIRES service mode page, if line spectrum is dominated by 1-2
lines then weaker lines may not affect cross-correlation (for calibration);
in that case use logs of the line intensities (--wl_log=true for
pipeline recipe crires_spec_wavecal).
</li>
<li>
Recall how I was confused about the flats getting brighter? In turns
out these are not dusk/dawn twilight flats but instead are using a lamp
and they increase the exposure level in each flat to handle non-linear
effects with a quadartic fitting function. So probably it'll be tough
for me to create master flats on my own and I'll want to use esorex :)
</li>
<li>
If I was going to do this manually more useful information here
(specifically that quadratic pixel fitting function):
https://www.eso.org/observing/dfo/quality/CRIRES/pipeline/recipe_calib.html
Also note that one should subtract darks of the same DIT (exposure time)
from the flat; also for making master darks (I didn't check the exposure
times for either!!)
</li>
<li>
Sounds useful: Recipe. All CRIRES science data are reduced using the pipeline recipe cires_spec_jitter. The recipe offers the following reduction steps: dark subtraction, correction for detector non-linearity, flat-fielding, image combination, spectrum extraction, wavelength calibration.
Also note that for nodding, dark subtraction is handled automatically with
the sky subtraction (pikachu waaat meme) -- it makes sense though because
the sky includes the dark current so when subtracting from the science frame
it removes the dark current and sky background.
</li>
<li>
NOTE: Thorium-Argon lamp provides lots of lines UP TO 2500nm (so that
won't likely be useful for calibration for us; in contrast sky lines
should be plentiful). Also N20 lines are recognized wavelength standards
beyond 3500nm; shorter wavelengths availabile in the HITRAN database (CFA)
but note no lines below 1650nm and weak lines 1650-2100nm -- so it sounds
like we can use the N20 lines as an independent check of calibration
using the sky line! There's also a CO gas cell and mentions NIST has
a line list (so I'm not sure if it's relevant for us but I would think so).
Note that the atmosphere contains N20 lines which affect L band (us),
although I'm not sure why we care for wavelength calibration whether the
line originates from the atmosphere vs the lamp...
</li>
<li>
Day 6 8:30am
</li>
<li>
Yesterday I spent a total of 3 hours over ~6 sessions. All of it was
reading the CRIRES webpages on eso or the user manual, though which
is sort of what I was trying to avoid as far as doing this as efficiently
as possible. Oh well. I've only spent a total of 10 hours 15 minutes on
this project so far. It'll be interesting to see how long it takes me
to make a paper totally solo. I'd guess somewhere between 100 - 1000 hours.
Since yesterday was rather hardcore and I need to put more time into
my other project + referee report + applications, I've decided to scale
this back. The goal will be to put in 0.5 - 1.5 quality hours per day
(note that the hours I'm tracking are not calendar hours as breaks don't
count; I'm not sure if that's in practice best since yesterday I skipped
lunch and didn't take breaks as a result...)
</li>
<li>
This first session I'll continue reading, but more ideally skim for
less relevant sections, the CRIRES user manual. I'm on page 82/129
and would like to finish it.
</li>
<li>
Note that the FITS file labelled SKY part way through the night
was likely for characterizing how much water vapor there was
in the sky (has AO and tracking off for it).
</li>
<li>
Birkby+ 2013 says they use order 17 and Table 8b of the CRIRES user
manual suggests using sky emission lines for order 17; good to know.
</li>
<li>
Okay so I finished the CRIRES user manula and returned to the cookbook
document. However, it referenced not just the CRIRES user manual but
also a CRIRES pipeline manual...So I took a look at the ESO manual
section again and I found the latest version of the CRIRES pipeline
manual but don't see a way to find historical versions. So I'll have
to hope things haven't changed too much. I'm now on p. 17/91 and it's
walking through how to use gasgano. So I guess I'll install and follow
along.
</li>
<li>
Hmmmm. The Gasgano page says that one should download it as part of the
instrument specfic software kit. Back on the pipelines page I can download
the latest kit version. But I wonder if this will be rather a problem
since this version is much later than what would've been used by Birkby+ 2013.
I found that I could replace the version number in the link URL to download
an older version of the source kit. Anyway I looked at Birkby+ 2013 again
and they used esorex. So I guess I'll just download the latest version
of the source kit but plan to do things manually / with esorex to make
sure I understand what's going on and that there's not an anachronism problem.
</li>
<li>
Automated crires-source kit installation failed because cpl is already
installed. I don't really feel like using gasgano that much so I'll
just leave this be for now. I'm now on p. 23/91 of the pipeline manual.
That's all for today (3 sets of 20 minutes = 11.25 hours total).
</li>
<li>
END (for now)
</li>
</ul>
</ol>