6.4.0: enhanced member number mapping and error detection

Headline feature 1: member number typo detection

Sorry, Evan, don't mean to pick on you!

Using fuzzy string search, USPSA Analyst can now detect probable typos in its member data. When two member numbers share a name, and those two member numbers are sufficiently close, Analyst will prompt you to fix it or ignore it.

Headline feature 2: multi-stage member number mapping

A small number of people in USPSA have more than two member numbers. Now Analyst can detect and handle them.

These updates pass all of my tests for member number mapping, but may not catch all cases, and the set of cases they don't catch may differ from earlier releases. Please let me know if you come across any.

Full changelog

Improvements

• Better wording for help text in various data entry dialogs.
• Behind-the-scenes recognized divisions settings, to ignore scores in certain divisions at certain matches.
  • Sorry, JJ, no more Open Elo from winning Open at Prod/PCC Nationals.
  • UI forthcoming.
• Change to non-stupid focus order when tabbing through fields on several screens.
• Some small rating calculation optimizations.
• Check mark when multiselecting in match cache dialog moved to left of row so you can see it happening.
• Member number mapping improvements as described above.
• Major match and L3 dataset exports are now in the release files.

Fixes

• Remove huge unintended slowdown in match downloads on configure-ratings page.

6.3.1: member number collision repair, bugfixes

The headline feature in this release is improved support for correcting various errors in the dataset. USPSA Analyst will prompt you to fix issues it detects, and now has additional support for correcting member number entry errors: typos, misplaced digits, and the like.

There are also some smaller features, plus fixes for issues found in 6.3.0:

• Fix loading bar in keep-history mode
• Fix slow match name loading for large lists of match URLs
• Shooter deduplication improvements in keep-history mode
• Search by member number in rater view
• Improve error messages and logging
• Tidy up some menus

6.3.0: usability improvements

uspsa-analyst-630-features.mp4

6.3.0 is all about usability and quality of life: lots of improvements to UI issues that had been bothering me for ages. The end result is a significantly more ergonomic tool.

Features

• Allow adding matches and editing hidden shooters from the rating view screen, and add the ability to save from the same screen.
• Start fetching matches immediately when adding uncached URLs to rating projects.
• Improve error messages for match download failures.
• When the application fails to download a match for a rating project, provide an option to remove failed URLs in the dialog shown by the popup message.
• Improve member number mapping algorithm.
• Show average history length in rater stats dialog.
• Show downloaded match or open Practiscore page when clicking on rows in rater configuration.
• Confirm exit from rating view screen, since recalculating large projects can take a bit.
• Multiselect mode for match cache dialog: when adding matches to a project from the cache, select multiple matches then confirm.
• Add manual member number mapping to project settings.
  • If the automatic member number mapping fails to detect a member number change for a shooter, you can give it a hint here.
• Add member number mapping blacklist.
  • If the automatic member number mapper maps two distinct shooters who share a name, you can tell it not to do that here.
• Improve error messages when adding URLs from parsed HTML.
• Improve dialogs wherever member numbers are entered.
• Improve match URL entry dialog.
• Add 'hidden shooters' feature: use shooters to calculate ratings, but block them from display, e.g. for building club or section ratings without clutter from traveling shooters.
• Add name entry field when creating new projects.
• Reorganize dropdown menus.

Fixes

• Fix broken member number mapping defaults.
• More links correctly display the 'click' cursor.

Internal stuff

• Build documentation updates.
• Scroll behavior fixes.
• Update to Flutter 3.3.10. Performance may be improved.
• Tweaks to internal benchmarking setup.
• Tests for member number mapping (automatic, manual, and blacklist).

6.2.0: more settings, improved shooter deduplication

I had hoped to drop a much bigger release (probably even 7.0), but database work has posed unexpected obstacles.

This release has some smaller changes instead, things that I tweaked or fixed while working on the DB project.

Features

1. More configurability! You can now change just about every number the Elo algorithm uses.
2. New defaults! Some experimentation with the new settings suggested some tweaks that would improve the quality of the output, so I made them. Please note that this is a breaking change. The new defaults are different from the old hardcoded values, so rating projects created before 6.2.0 will yield different results unless you make the changes in the 'rating changes' section below.
3. Improved loading times! The match cache begins loading in the background immediately on app start, so you'll save a second or two on the way to common features like viewing a match from the cache or opening a rating project.
4. Improved calculation times! I want to address the lack of multithreaded calculation at some point in this project, but until then, I saved substantial amounts of time by updating the loading bar once per five matches processed rather than for every match processed.

Fixes

1. Improved shooter deduplication, thanks to Mel Rodero.
2. Added tests for same.
3. Some general code cleanup and maintainability improvements.

Rating Changes

The new default settings have several interrelated effects. The reduction in K factor from low-error predictions is smaller, so ratings are slightly more volatile at the top end, while the increase in K factor from high-error predictions is larger, so ratings find their true value still more rapidly.

The new K and probability base settings make ratings a bit fuzzier. By stage mode provides the most datapoints and is therefore the most accurate basis for ratings, in my opinion, but have more noise in them—the match winner doesn't have to win every stage. A smaller probability base helps smooth out that noise by reducing the impact of narrow misses around your expected performance.

To make a 6.2.0 rating project yield the same values as 6.1.0, you'll need to change the default settings to the old values:

• K factor: 60
• Probability base: 10.0
• Error-aware K zero value: 0
• Min threshold: 100
• Max threshold: 100
• Lower multiplier: 0.5
• Upper multiplier: 2.0

Upcoming Work

The next big item on my list is evolutionary parameter tuning, picking some 2022 majors as calibration matches. (Nationals, probably, and maybe an area match or two.)

Before the 2023 season starts in earnest, I'd like to have that work fully squared away.

6.1.0: match cache improvements and more

6.1.0 has a ton of new features, mostly involving new ways to view results and research the historical causes of rating changes.

General changes

• Add match cache search/management dialog on main page: add single matches, view matches, or remove matches from the cache.
• Various match cache improvements.
• Fix some match parsing issues.
• Swap to 'this is a link' mouse pointer on a larger number of clickable areas.

Ratings

• Clicking on a stage name or graph segment in the shooter stats dialog takes you to the relevant match results.
• Show estimated percentage in prediction error bars, rather than raw predictions.
• Add button to view match results for matches in a ratings dataset.
• Add a 'parse practiscore page source' option for adding matches to a rater dataset: follow the instructions in the help dialog to view page source, let Analyst do the rest.
• Retain original member number, and link shooter names to uspsa.org shooter pages in various places.
• Clicking the 'histogram' text in rater stats dialog switches to quartile view.
• Rater stats dialog calculates stats for the currently-filtered shooters only.
• Improvements to 'processing scores' portion of loading screen.
• Inverse place points series rater slightly simplified: you get 1 point for each person you beat

6.0.0: points series and predictions!

6.0.0 sees a bevy of changes. Foremost is predictions! The rating engine uses Monte Carlo simulation to generate a range of expected performances for each shooter, then does some voodoo to turn them into approximate percentages. The algorithm correctly places shooters into the 95% confidence interval about 80% of the time, and into the 68% confidence interval about 70% of the time.

Other changes

• Alternate rating systems: OpenSkill (takes too long to converge in its present form), points series (for scoring club or match points series)
• New icons
• Huge speed improvements (again)
• Match blend defaults to 0.3, error-aware K defaults to on (they work well)
• Import and export start in a slightly more sensible directory
• CSV exports in more places
• Sort ratings by first or last name

5.2.0: error tracking and compensation

This release has some serious improvements to the math, some moderate improvements to the speed, and some cool improvements to the UI.

TL;DR Features

• Match blend now affects both place and percent
• Track error in ratings
• Error compensation in rating math
• Show error on shooter chart
• Chart expands on larger screens
• More speed improvements
• Classification is now displayed on rating pages
• Max Days now defaults to 365
• Rating pages now have a sort dropdown
• Import/export file pickers now start in the USPSA result viewer directory

Error and error compensation

A major weakness in earlier versions of the math was the difficulty in balancing quick placement of shooters entering the dataset, and stable placement of shooters with a long history. That problem is largely solved with the introduction of an error measure, which allows the algorithm to track the accuracy of its predictions over time.

Tracking is only half the story: the algorithm also uses its error for a given shooter to adjust his K factor. Shooters with high error get an upward adjustment to their K factor, on the theory that it will bring them toward their correct rating more quickly. Shooters with low error receive a downward adjustment to their K factor, on the assumption that their rating is already nearly correct if the algorithm is predicting their results correctly.

Consider some examples:

A good but inconsistent shooter. Since his rating is up and down, his error remains moderate and his rating remains fairly fluid.

Nils Jonasson, who the system has accurately placed on the basis of continued strong performances.

Christian Sailer in Limited 10. Although he only shot one match in the division, his extreme overperformance relative to the starting rating for U yielded a high error, which means his rating adjusted quickly to near its correct level.

A shooter who plateaued, then improved. His error grows smaller toward the end of the plateau, then increases again as he starts improving, allowing his rating to improve more quickly.

5.1.0: Elo rater improvements

Elo rater

• Improved chart in shooter stats modal
  • Now shows absolute rating instead of rating delta
  • Mousing over the rating history line highlights the associated rating event in the history list
• Match score blend feature: in by-stage mode, mix in a proportion of a shooter's match score into his stage scores, to help limit the impact of bad stages
  • WIP, doesn't work very well at solving the stated issue yet
• Tremendous improvements in speed (rates all L2+ matches since 2020 in about 25 seconds)
  • There's more meat on the bone, too, but the low-hanging fruit is mostly taken car of
• Match cache loading now has a progress bar

Fixes

• Result viewer no longer lists all stage scores as 'null' when exiting what-if mode
• Result viewer retains selected stages when leaving what-if mode
• Elo rater on Linux no longer gets stuck on one tab after a short time

5.0.0: Elo rating system

Elo Rating System

This release contains desktop applications for Windows and Linux (simply extract and run). In addition to the regular match result viewer, these applications also contain an Elo rating engine for USPSA matches. To use it, pick the appropriate option from the main menu, add matches to rate, and hit the 'advance' button.

For (much) more, see README-Elo.md.

Result Viewer Changes

• Sort times of 0 as infinity, when sorting by time.
• [Web only] Move to a self-hosted CORS proxy for loading from Practiscore, which reduces load times and also solves the issue where Heroku was going to kill off my free dyno later this year.

5.0.0-rc1: Elo rater release candidate

See beta2 for initial notes on the Elo rater.

This release is the first 5.0.0 release candidate, and contains some quality of life improvements as well as a few features.

QoL/Fixes

• Add more confirmation on destructive actions in rater configuration.
• When adding match URLs in rater configuration, add a list at the top.
• Show a count of match URLs in rater configuration.
• Move some actions to overflow menu in rater configuration.
• Add 'close' button on shooter stats dialog.
• In match results view, on Comstock and Virginia Count stages, sort times of '0' to the end when sorting by time.
• Massive improvements to performance/memory use.

Features

• Sort button for match URL list in rater configuration: sorts cached matches by date descending, and puts unseen URLs at the top of the list.
• "Last seen" filter in ratings view: hide shooters whose last activity exceeds a certain age.
• Revamp trend measure: now it's the total change in rating over the trend window.
• Add manual shooter aliases, for the case where a shooter switches member numbers (i.e., from an annual membership to a lifetime membership), and also enters their name differently across the boundary. cf. Max Michel, who entered his name as Max Michel Jr. in 2020, but switched to 'Max Michel' sans Jr. before getting his lifetime member number.