Skip to content

Pupil Capture, Player, and Service release (updated)
Compare
Choose a tag to compare
@papr papr released this 11 Jun 09:08
· 1435 commits to master since this release
v2.0
04cc12c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

We are pleased to announce the release of Pupil Core software v2.0!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

With v2.0 and beyond, we will be focusing on making Pupil Core more core.

Our vision for Pupil Core emerged early on: we wanted to make eye tracking hardware and software that was accessible. Accessibility is a useful keyword for us because it encourages us to think about the diverse group of researchers that use Pupil. Pupil Core strives for the following:

  • Transparency: Open source is key here. Enable researchers to introspect into the algorithms. Be explicit about the methods and biases built into the software. Inspire others to make improvements and catch bugs.
  • Modularity: Make it easy for users to develop and run their own custom code - plugin architecture.
  • Connectivity: Make it easy to connect to other software - network API.
  • Diversity: Pupil core is flexible. Pupil Core strives to accommodate a diverse range of research and applications. Pupil Core enables researchers who want to use it as a desktop app with GUI, researchers who want to integrate with other software and sensors via Network API and plugins, and developers who want to tinker with the source code directly.

Over the years, Pupil Core software grew organically to support the community, new hardware, and novel applications. With v2.0 took a few steps back to look at the big picture and to think about the future of Pupil Core. If we want Pupil Core to continue to be a useful platform for researchers and developers, then we need to boil it down to its essentials.

How are we making Pupil Core more core? By focusing on making essential (or core) parts of Pupil Core work really well. Here are the steps we have taken with v2.0:

  1. Restructuring the core pipeline: We have made changes to the application architecture that allows it to run multiple pupil detectors in parallel as well as easily add custom gaze mapping algorithms via user plugins.
  2. Removing hard to maintain features: We had to make some tough decisions. We decided to remove audio capture and native Intel RealSense support. These components were too much work to maintain.
  3. Simplifying the user interface: Over the years we have added many features and the UI has grown organically in the process. We have decided to streamline and improve the UI. Be it merging of menus or more sane default settings, we felt that new users need a bit more help getting started with Pupil Capture and Player.

Improvements

Parallel 2D and 3D Pupil Detection

To allow for using different pupil detection algorithms, we have decoupled our existing pipeline of 2D and 3D detection. Instead of running either of those, both will now run as separate plugins with only minimal overhead. This will enable researchers (including our own R&D team) to test, optimize and replace parts of the pipeline much more conveniently in the future.

Additionally this change has allowed for the following improvements to other parts of the system:

Pupil Diameter Timelines - #1854

With 3D pupil data always available we can always display the meaningful 3D pupil diameter in the timelines. Additionally, we added outlier removal and included the range for a more expressive visualization.

Blink Detection - #1900

With 2D pupil data always available we can run a more reliable blink detection in all circumstances.

Calibration Choreography and Gazer Plugins

By far the biggest change in this release was the refactoring of calibration and gaze mapping part of the core pipeline into separate, extendable components that are easy to swap with custom implementations.

Calibration Plugin in Pupil Capture

When running Pupil Capture, in the plugin list you can find the unified Calibration plugin, which is responsible for configuring both the calibration method (with the Choreography selector), and the gaze mapping method (with the Method selector).

Extending Calibration and Gaze Mapping with User Plugins

Now users can implement their own calibration routines and gaze mapping methods. These will automatically show up in the Choreography and Method selectors.

To do this, users must subclass CalibrationChoreographyPlugin and GazerBase respectively. See the Developer Notes section below for a code example.

Improved Installation Workflow on Windows - #1853

We have wrapped Pupil in a Windows Installer package (MSI) in order to simplify the Windows workflow.

By default, all 3 apps (Capture, Player, and Service) will be installed in C:\Program Files (x86)\Pupil-labs\Pupil v<version>\. All apps will also get a start-menu entry, making it much easier to open Pupil. Installed versions of Pupil can be removed in the Windows Uninstall Settings.

New versions of Pupil will be installed alongside older versions and you can choose which to start. Switching versions will still overwrite your user settings as previously.

Changes

Fixation and Blink Detection by Default - #1870

In order to make Pupil Core even more useful with out-of-the-box settings, we have enabled the Fixation Detector and Blink Detector by default.

Simplified System Graphs - #1869

We have removed the configuration menu for the System Graphs plugin in order to simplify the user interface. The optional display of live graphs for the pupil diameter has been removed as well.

The confidence graphs will now always display the confidence of the 2D pupil detection, as this is a better indicator for correct positioning of the Pupil Core eye tracking headset.

Single Marker Calibration: Physical Markers - #1897

The single marker calibration is mostly intended for calibration with physical (printed) calibration markers, as opposed to the screen marker calibration. For sake of clarity, we changed the default mode for the single marker calibration to Physical Marker (previously this was called Manual).

Removed Manual Marker Calibration

With the restructuring of the calibration workflow, we have removed the deprecated Manual Marker Calibration. The Single Marker Calibration is a more general version and additionally allows for VOR-based choreographies. If you previously used the Manual Marker Calibration, try the Single Marker Calibration in Physical Marker mode (the new default)!

Parallel Pupil Detector Visualization - #1873

We have changed the color of the 2D ellipse visualization in all eye videos to blue in order to differentiate between 2D and 3D detection. You will now always see 3 ellipses on the eyes when the visualization is enabled:

Color Description

  • blue: 2D pupil detection result
  • red: 3D pupil detection result
  • green: 3D eye model

Flip Eye 0 Display - #1844

The eye 0 (right eye) camera sensor is physically flipped, which results in an upside-down eye image for the right eye. This is by design and does not negatively affect pupil detection or gaze estimation. However, the upside-down eye 0 image repeatedly led users to believe that something was broken or incorrect with Pupil Core headsets. We flipped the eye 0 image now by default to better match user expectations with a right-side-up eye image.

Deprecated Audio Capture - #1868

Recording synchronized audio on 3 different operating systems (macOS, Linux, Windows) has always been a challenge for Pupil Capture. (See our motivation section above.)

Unified Network API Plugin - #1881

The previous plugins Pupil Remote and Frame Publisher were commonly used to interact with Pupil Core's Network API. For the sake of simplicity and clarity, we merged both plugins into the new Network API plugin.

Removed Blink Detector Visualization - #1901

The visualization of the blink detector (brief dark flashing on the screen) could interfere with other plugins and cause visual bugs. We have removed this feature.

Bug Fixes

  • Fixed Player crashing when exporting while computing gaze history - #1843
  • Fixed Player crashing when disabling the Surface Tracker while exporting - #1848
  • Fixed wrong default camera selection in some cases - #1857
  • Fixed a rare crash in the Surface Tracker - #1879
  • Fixed deprecated RealSense cameras showing up in the camera selection - #1877
  • Fixed that offline gaze mappers without data would show as completed - #1889
  • Fixed corrupted gaze data when deactivating one of multiple gaze mappers - #1888
  • Fixed new offline pupil data not being correctly announced sometimes - #1890
  • Fixed a few small issues in offline pupil detection - #1899
  • Fixed problems with opening the macOS bundles on newer versions of macOS

Developer Notes

Changed Requirements

Scikit-learn

We have added scikit-learn as a dependency for Pupil.

pip install scikit-learn

PyAV - 1908

We updated the PyAV dependency to v0.4.5, which includes some bug fixes. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/PyAV

On Windows, download and install the latest wheel from GitHub.

Changed data format

Both 2D and 3D pupil detector now publish their results independently of each other. To differentiate between them, we have added the origin to the topic. For a binocular setup, you will now get 4 different pupil topics:

  • pupil.0.2d
  • pupil.0.3d
  • pupil.1.2d
  • pupil.1.3d

This means that there can now be multiple pupil datums for one eye for the same timestamp (2D and 3D). When grouping by the entire topic however, the timestamps are still guaranteed to be unique.

Restarting Unique Plugins - #1871

We have added the previously missing feature to restart plugins with by_class uniqueness via notifications.

Recording Version 2.2 - #1885

The info.player.json meta-version has been bumped to 2.2 will be incompatible with Pupil Player version v1.

Allow specifying ZMQ HWM option - #1895

We added the option to start the eye processes with custom values for the ZMQ High Water Mark (HWM) via notification.

Custom Calibration Plugins

Below is an example with minimal implementation for a custom HMD 3D calibration choreography and gaze mapping method:

# To implement a custom calibration or gaze mapping method from scratch, subclass GazerBase and CalibrationChoreographyPlugin
from gaze_mapping.gazer_base import GazerBase
from calibration_choreography import CalibrationChoreographyPlugin

# Another option is to modify existing implementations by subclassing existing implementations
from gaze_mapping import Gazer3D
from calibration_choreography import ScreenMarkerChoreographyPlugin


class CustomGazer3D(Gazer3D):
    # Must be unique across all subclasses of GazerBase
    label = "Custom 3D Gazer"


class CustomChoreographyPlugin(ScreenMarkerChoreographyPlugin):
    # Must be unique accross all subclasses of CalibrationChoreographyPlugin
    label = "Custom Calibration"

    @classmethod
    def supported_gazer_classes(cls):
        return [CustomGazer3D]

Release Note Updates

17.06.2020 - v2.0-175

  • Added compatibility for the HMD-Eyes VR integration
  • Fixed an issue that prevented the validation from working correctly

24.06.2020 - v2.0-177

  • Fixed an issue that prevented using the head pose tracker and creating new annotations in Player - #1927

25.06.2020 - v2.0-182

  • Fixed an issue that prevented Pupil Service from starting - #1928

Downloads

Officially minimum supported operating system versions for the bundles:

Ubuntu: 16.04
macOS: 10.13 (High Sierra)
Windows: 10

To open the RAR-archive on Windows, you will need a decompression software, e.g. WinRAR.

Assets 5
Loading