Auto Picker #19
Merged
Auto Picker #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Still needs more work for GPS info. The NMEA strings written on these files don't seem to be complete.
This is to finalize the scripts for Hills et al. (2020) to be submitted to the Radioglaciology Annals volume. A total of 6 attenuation methods implemented: Method 2 -- Jacobel et al. (2009) Method 3 -- Schroeder et al. (2016a, 2016b) Method 5 -- MacGregor et al. (2014); Matsuoka et al. (2010) Method 6a -- Depth-resolved (regression similar to method 1) Method 6b -- Depth-resolved (regression similar to method 3) Method 7 -- Secondary reflection as in Christianson et al. (2016)
Accommodate multiple files and multiple bursts
Following Brennan et al. (2013) and Nicholls et al. (2015) calculate the vertical motion using a moving window and correlation coefficient calculated at subsequent depths in two acquisition profiles. Vertical ice motion between acquisitions can be seen for both internal reflections and the bed reflector.
Uses the correlation coefficient between two ApRES profiles to determine the relative motion between acquisitions.
If the gps data onto which the interpolation will be done has longitude notation different than the radar longitude, then raise an error. We were having problems with one being written from 0 - 360 and the other from -180 - 180.
Center the depth profile with each point now representing the center of the window rather than the top.
There were too many options in the plot command which made it confusing. I cut some out and hopefully made the help file more clear as to how one could plot the spectral density for each trace using Joshua's function.
Following Kingslake et al. (2014) calculate uncertainty with a noise phasor that has random phase and magnitude equal to the median of the measured phasor.
Line to load the time step from the header file was looking for the time window as 'WINDOW' when a later line has 'AMPLITUDE WINDOW' so it was loading that number instead.
These files are mostly the same as the ImpDAR matlab files but commonly have the data named as 'filtdata', 'interp_data', 'migdata', etc.
There were too many variables named similar things, so I made it a bit more clear.
Removed 1e6 on the depth vector
This routine asks for some picked points on the left side of the image and will spit out picks across the image for each.
The autopicker was not working nicely with the normal picker. That is, when switching back and forth, the program would break. This was because of an indexing issue with the pick number. Fixed.
…of .mat files, allow these to be loaded into ImpDAR
|
I'm going to hold off on this for a bit. A number of people are using the devel version since when new loaders are needed I put them on there to avoid problems with master. Let's plan on merging devel into master and having to test that in December. After that merge, we should revisit this. |
|
Yeah that is no problem. Maybe that will give me a chance to clean things up. Also, it looks like I need to look into this 'coveralls' thing. That is new to me. |
This is in regard to the adaptive horizontal filter. The scaling function that is used to keep the filter from changing the deeper data is much more complicated than what I would have used. Considering simplifying.
|
Yeah, don't worry too much about coveralls. It will say that checks have failed if test coverage decreases, but I think we should prioritize getting features in, and if you can write useful tests then that is great too. |
This was overly complicated and different from that being used in hfilt. I just copied from hfilt, and cleaned up some of the code. I also added optionality for setting the window_size from the command line function.
|
I added some commits here that address #20. Still no rush on pushing this forward. |
|
No tests run due to problems with travis.com billing. Going to put this all in, then get tests running. |
dlilien
added a commit
that referenced
this pull request
Nov 10, 2021
* Dual axes (#14) * Add support for dual axes (TWTT and depth) * Fix very opaque error: import order of h5py and netCDF4 matters. See comment on Unidata/netcdf4-python#653 by arthur-e * Fix syntax error * Fix bug in variable-rho NMO exposed by update to scipy * Allow switches between old StoDeep matrices when picking (#15) * Give users a string representation of RadarData and Picks objects Guard against None in t_srs * Docs go to links, ref. bluesystems documentation * Prevent file dialation by incorrect type casting for GSSI * When you delete picks, resuming picking resumes from where you deleted to * BSI can handle bad GPS data * Fix missing trace num on bsi * Sequentialize to reduce peak memory (#18) * Sequentialize to reduce peak memory Operations from the command line are now mostly sequential. This should reduce peak memory usage for most operations. Exceptions are concatenating and loading gecko or osu files. * Make the list copy python 2/3 compatible * Fully test processing, with more options to concatenate profiles with differing picks * Move Pulse EKKO partitioning into the load() function As per issue #16, the PE project files were inconsistently handled because the partitioning was being done in load_and_exit(). Move that part of the function into load(). Also, change the loop so that the load() function can handle a mix of project files and standard PE files. Co-authored-by: Benjamin Hills <bhills@uw.edu> * Fix order of operations error with W, E with BSI Had resulted in an error of 0.66666 in longitude * Minor bug fixes * Travis forced migration to .com, update badge * Fix some issues with version-dependent colors, segy * Fix problem where on some versions strings are already decoded for BSI * Fix second BSI decode issue * Try to accept skbuild errors on 2.7 * Try to fix skbuild on 2.7 * Update readme for Python2.7 EOL * Begin switch to github actions Travis is not running on .com due to lack of credits--we are not going to pay. * Install impdar before testing * Auto Picker (#19) * First commit for DeLoRES reader Still needs more work for GPS info. The NMEA strings written on these files don't seem to be complete. * Slight change to Pulse Ekko reader for OSU group. * Change for Pulse Ekko GPS files with more than only GGA strings. * Slight change for the GSSI reader for data from the OSU group. * Add a reader for the OSU HF data * A few small changes for OSU data, interpolation, plotting * Add a reader for MALA RAMAC data * Allow alternative data variable names for Olaf Matlab files. * Update and test all of the analysis scripts. This is to finalize the scripts for Hills et al. (2020) to be submitted to the Radioglaciology Annals volume. A total of 6 attenuation methods implemented: Method 2 -- Jacobel et al. (2009) Method 3 -- Schroeder et al. (2016a, 2016b) Method 5 -- MacGregor et al. (2014); Matsuoka et al. (2010) Method 6a -- Depth-resolved (regression similar to method 1) Method 6b -- Depth-resolved (regression similar to method 3) Method 7 -- Secondary reflection as in Christianson et al. (2016) * Change analysis file names. * Update error in attenuation calculations. * Update load script for ApRES Accommodate multiple files and multiple bursts * Add a range differencing function for ApRES. Following Brennan et al. (2013) and Nicholls et al. (2015) calculate the vertical motion using a moving window and correlation coefficient calculated at subsequent depths in two acquisition profiles. Vertical ice motion between acquisitions can be seen for both internal reflections and the bed reflector. * Add a range differencing function to the ApRES processor. Uses the correlation coefficient between two ApRES profiles to determine the relative motion between acquisitions. * Add option for external depth array in ApRES range differencing. * Add a warning on interpolation If the gps data onto which the interpolation will be done has longitude notation different than the radar longitude, then raise an error. We were having problems with one being written from 0 - 360 and the other from -180 - 180. * Some small fixes for typos in hcrop * Add error to ApRES coherence calculation. * Update the ApRES range function to output uncertainty * Small change in ApRES range differencing function Center the depth profile with each point now representing the center of the window rather than the top. * Some changes to Joshua's spectral density plotting. There were too many options in the plot command which made it confusing. I cut some out and hopefully made the help file more clear as to how one could plot the spectral density for each trace using Joshua's function. * Update specdense plotting to satisfy Travis. * Add a function for calculating the phase/range uncertainty. Following Kingslake et al. (2014) calculate uncertainty with a noise phasor that has random phase and magnitude equal to the median of the measured phasor. * Typo in Pulse EKKO load script. Line to load the time step from the header file was looking for the time window as 'WINDOW' when a later line has 'AMPLITUDE WINDOW' so it was loading that number instead. * Add a load script for the St Olaf matlab files. These files are mostly the same as the ImpDAR matlab files but commonly have the data named as 'filtdata', 'interp_data', 'migdata', etc. * Changes to naming convention for ApRES uncertainty There were too many variables named similar things, so I made it a bit more clear. * Changes to the crop function for pretrig * Change some notation in the ApRES scripts. * Typo in Kirchhoff Migration Removed 1e6 on the depth vector * Add an autopicking routine. This routine asks for some picked points on the left side of the image and will spit out picks across the image for each. * Allow auto picks from any point on the image * Fixed some bugs with the sequence of the autopicker. The autopicker was not working nicely with the normal picker. That is, when switching back and forth, the program would break. This was because of an indexing issue with the pick number. Fixed. * Some of the CReSIS data products are saved as an alternative version of .mat files, allow these to be loaded into ImpDAR * Add a try/except line for h5py import so that Travis will be happy. * Created Issue #20 This is in regard to the adaptive horizontal filter. The scaling function that is used to keep the filter from changing the deeper data is much more complicated than what I would have used. Considering simplifying. * Simplify the scaling function for ahfilt This was overly complicated and different from that being used in hfilt. I just copied from hfilt, and cleaned up some of the code. I also added optionality for setting the window_size from the command line function. * Change the adaptive hfilt test to make Travis happy. * A mess of a PR (mostly ApRES scripts) (#21) * First commit for DeLoRES reader Still needs more work for GPS info. The NMEA strings written on these files don't seem to be complete. * Slight change to Pulse Ekko reader for OSU group. * Change for Pulse Ekko GPS files with more than only GGA strings. * Slight change for the GSSI reader for data from the OSU group. * Add a reader for the OSU HF data * A few small changes for OSU data, interpolation, plotting * Add a reader for MALA RAMAC data * Allow alternative data variable names for Olaf Matlab files. * Update and test all of the analysis scripts. This is to finalize the scripts for Hills et al. (2020) to be submitted to the Radioglaciology Annals volume. A total of 6 attenuation methods implemented: Method 2 -- Jacobel et al. (2009) Method 3 -- Schroeder et al. (2016a, 2016b) Method 5 -- MacGregor et al. (2014); Matsuoka et al. (2010) Method 6a -- Depth-resolved (regression similar to method 1) Method 6b -- Depth-resolved (regression similar to method 3) Method 7 -- Secondary reflection as in Christianson et al. (2016) * Change analysis file names. * Update error in attenuation calculations. * Update load script for ApRES Accommodate multiple files and multiple bursts * Add a range differencing function for ApRES. Following Brennan et al. (2013) and Nicholls et al. (2015) calculate the vertical motion using a moving window and correlation coefficient calculated at subsequent depths in two acquisition profiles. Vertical ice motion between acquisitions can be seen for both internal reflections and the bed reflector. * Add a range differencing function to the ApRES processor. Uses the correlation coefficient between two ApRES profiles to determine the relative motion between acquisitions. * Add option for external depth array in ApRES range differencing. * Add a warning on interpolation If the gps data onto which the interpolation will be done has longitude notation different than the radar longitude, then raise an error. We were having problems with one being written from 0 - 360 and the other from -180 - 180. * Some small fixes for typos in hcrop * Add error to ApRES coherence calculation. * Update the ApRES range function to output uncertainty * Small change in ApRES range differencing function Center the depth profile with each point now representing the center of the window rather than the top. * Some changes to Joshua's spectral density plotting. There were too many options in the plot command which made it confusing. I cut some out and hopefully made the help file more clear as to how one could plot the spectral density for each trace using Joshua's function. * Update specdense plotting to satisfy Travis. * Add a function for calculating the phase/range uncertainty. Following Kingslake et al. (2014) calculate uncertainty with a noise phasor that has random phase and magnitude equal to the median of the measured phasor. * Cause upload to PyPi (1.0.1 and 1.0.2 are sadly missing) * Fix the OSU load script Very minor typo to be fixed on the file name variable. * fix nan issue in load * Fix test naming for coveralls * Again, coveralls * Switch to codecov * Learning yml syntax... * Try testing with tox as recommended * More bad syntax * Go back to using pytest * No really, pytest not unittest * Try an optional dep * More syntax * Try pyqt5 wheel * Bad qt version? * Nope, really newer qt * Try with xvfb * Syntax again * Matplotlib deprecated no radius for picker * h5py and segyio required, qt5, seisunix, gdal optional * Rebadge * Use github version of seisunix * Use github version of seisunix * Do not let gh check seisunix with flake8 * Yes to install for su * Try to skip known problems * Remove bash from yml * And try again * Split testing for GUI/non * Add in GDAL * Exclude gdal on non-linux * Try newer gdal * Try to make Windows work * Try to make Windows work with overflow * Start testing PyPi upload * Jobs need runs-on or uses... * Just one job. * Many linux? * Was entrypoint permissions the issue for manylinux? * Try to fix twine issues for manylinux * And try a MacOS build * Help MacOS find fortran * 3.10 just is not ready yet * Fix success checks * Try to diagnose problem uploading to codecov * Continue diagnosis * Try skipping pytest-cov * reorder args * Dodge bad gdal on pip for 3.9 and 3.0.4 * Finalizing bug fixes before merge into master * Try to fix mac build * Dont fail for codecov * All set for merge, just skip building tarball extra times Co-authored-by: Benjamin Hills <bhills@uw.edu> Co-authored-by: Ben Hills <benjaminhhills@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We probably want to deal with #18 before this one, but I am opening it now. Let me know if that is not proper etiquette, I am still getting used to this stuff.
This is work that I did on the gui this summer to implement an autopicker. The current state is not finished (hence remaining on devel branch), but there is some functionality that could be useful to someone possibly. Also, this is not a priority of mine right now, so I figured I would submit a pull request before I forget about it. The idea is to select one point on the radar image and it will automatically pick that layer across the extent of the image. As of now, the picking algorithm is the same as the standard picker, which goes trace-by-trace to isolate the wavelet that is nearest to that of the previous trace.
I haven't thoroughly reviewed the changes, but at least Travis is happy. Eventually someone should probably clean up this work as well as the rest of the GUI code. Definitely feel free to put this on hold if you think we should put more work into the gui before moving this into devel.