Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding download and locale filtering options #299

Merged
merged 4 commits into from
Oct 6, 2020
Merged

Conversation

sffc
Copy link
Member

@sffc sffc commented Oct 5, 2020

Key changes in this PR:

  • Changes CldrPaths from a struct to a trait object
  • Adds a second implementation of CldrPaths that downloads CLDR JSON into the user's cache folder, as determined by dirs::cache_dir()
  • Adds the ability for the exporter to filter data by locale when building up the file structure
  • Adds options for CLDR tag and locale includelist into the icu4x-cldr-export binary tool

I added the download feature because:

  1. For the testdata PR (Add top-level testdata crate #296), I wanted reproducible results with minimal configuration, and I thought downloading a zip file corresponding to a GitHub tag would help.
  2. The developer experience might be nicer if you don't need to clone all the CLDR JSON repos, risk that they get stale, etc.

The locale filtering feature is bare-bones for now. We can upgrade it once we figure out how we are resolving locale negotiation in data loading (#173).

@sffc sffc requested a review from zbraniecki October 5, 2020 10:31
@jira-pull-request-webhook

This comment has been minimized.

@sffc sffc marked this pull request as ready for review October 5, 2020 10:55
@sffc sffc requested a review from a team as a code owner October 5, 2020 10:55
@coveralls
Copy link

coveralls commented Oct 5, 2020

Pull Request Test Coverage Report for Build e109caa49f2200845de6e2535ab9e1d6396858b2-PR-299

  • 72 of 184 (39.13%) changed or added relevant lines in 10 files are covered.
  • 17 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.6%) to 77.795%

Changes Missing Coverage Covered Lines Changed/Added Lines %
components/cldr-json-data-provider/src/transform/dates.rs 0 2 0.0%
components/fs-data-provider/src/manifest.rs 2 4 50.0%
components/cldr-json-data-provider/src/transform/plurals.rs 0 3 0.0%
components/cldr-json-data-provider/src/transform/mod.rs 1 5 20.0%
components/fs-data-provider/src/export/fs_exporter.rs 10 15 66.67%
components/cldr-json-data-provider/src/cldr_paths.rs 2 8 25.0%
components/cldr-json-data-provider/src/download/cldr_paths_download.rs 11 22 50.0%
components/cldr-json-data-provider/src/download/io_util.rs 46 66 69.7%
components/cldr-json-data-provider/src/download/error.rs 0 23 0.0%
components/fs-data-provider/src/bin/icu4x-cldr-export.rs 0 36 0.0%
Files with Coverage Reduction New Missed Lines %
components/fs-data-provider/src/manifest.rs 1 50.0%
components/pluralrules/src/error.rs 3 33.33%
components/pluralrules/src/data.rs 6 82.22%
components/pluralrules/src/operands.rs 7 83.54%
Totals Coverage Status
Change from base Build d3a6cd1651e9720fbd883cfa44d78906586213ad: 0.6%
Covered Lines: 2964
Relevant Lines: 3810

💛 - Coveralls

zbraniecki
zbraniecki previously approved these changes Oct 5, 2020
Copy link
Member

@zbraniecki zbraniecki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! Thank you!

/// The fields should be Ok if present. They default to Err when not present.
pub trait CldrPaths {
fn cldr_core(&self) -> Result<PathBuf, Error>;
fn cldr_dates(&self) -> Result<PathBuf, Error>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need to return PathBuf? Usually in such cases we use Path (just like we'd return &str not String).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you either have an owned PathBuf or a borrowed &Path. If I return &Path, then someone has to own it.

I figured it was fine to return PathBuf because the only call site immediately takes it and starts appending more pieces to it, so if we returned &Path, we'd need to call .to_path_buf() anyway.

/// https://github.com/unicode-cldr/cldr-core/tags
pub fn from_github_tag(github_tag: &str) -> Self {
Self {
cache_dir: dirs::cache_dir().unwrap().join("icu4x").join("cldr"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unwrap -> expect

#[non_exhaustive]
#[derive(Debug)]
pub enum Error {
JsonError(serde_json::error::Error),
IoError(std::io::Error, std::path::PathBuf),
MissingSource(MissingSourceError),
Download(Box<dyn error::Error>),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason you don't want to use DownloadError here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was either use a box or make this Download(DownloadError) and protect that enum variant with #[cfg(feature = "download")] everywhere it is used in this file. I had a slight preference for a box in order to reduce the number of places where I needed #[cfg(feature = "download")]. Do you have a preference?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I made this change.

@@ -1,12 +1,16 @@
use std::error;
use std::fmt;

#[cfg(feature = "download")]
use crate::download::error::Error as DownloadError;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since download is optional, would it be easier to maintain to have the From in download::error instead of here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, but I still need to put the Display and std::error::Error implementations in this file, so it seems odd that From would be in a separate place. It's easy to put it in this file, and I protect it with #[cfg(feature = "download")].

Copy link
Member

@zbraniecki zbraniecki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@sffc sffc merged commit 33847df into master Oct 6, 2020
@sffc sffc deleted the dp-improvements branch October 6, 2020 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants