Update documentation #81

sadikovi · 2018-04-10T09:19:19Z

This PR updates crate documentation and makes some very minor changes, either in code or comments.

Following is done:

Updated docs, so now we can generate more or less usable docs with cargo doc. I learned that it is good to run cargo doc --no-deps, to generate docs for this crate only when developing.
Added small example in README + usage + link to documentation.
Added comment for column/reader.rs tests to give an overview on how setup works.

I would encourage to view changes as generated documentation as well as code. You can check out this branch and run cargo doc --no-deps --open to open documentation in your browser.

Closes #58
Closes #77
Closes #78

sadikovi · 2018-04-10T09:37:25Z

@sunchao Could you review this PR? The goal of this work is having documentation/some starting point for people who want to read Parquet files.

Publishing docs

I added badge in README that points to docs.rs for parquet (it is broken currently). I thought about this and I think that publishing on docs.rs is arguably the best way to do it right now, which is what you also mentioned on the issue #78 .

My main points are:

No setup required, it is sort of generated automatically as we release.
Most of the people publish on docs.rs.

What stops us from doing this currently is generating parquet.rs file. Though I understand why you would want to have it generated (this is how it works right now), I would suggest that we actually commit the parquet.rs as part of the repository alongside parquet.thrift.

When we need to update format, we will create a special PR that updates parquet.rs only and potentially fixes code to work with updated format. We will document or have a script that would generate parquet.thrift -> parquet.rs and will run manually when we decide to update the format.

Reasons for committing parquet.rs into repository:

Fixes documentation build.
Simplifies travis setup, and speeds up build time (it is not really a big deal, tests run reasonably fast anyway).
parquet.rs does not change often, actually only changes when we change format.
Rarely thrift repo may be broken, so in this case we would not notice the impact of those breaking changes.
Assumption that parquet.rs is generated identically regardless of OS.
Helps other people to import the crate, without requirement of setting up thrift (and we would only use it to generate single file), all they would need is just adding parquet to their Cargo.toml.

Let me know your thoughts, as you are more familiar with overall setup and parquet.thrift generation. If so, I will be more than happy to open other pull requests to update the code.

coveralls · 2018-04-10T09:48:29Z

Coverage decreased (-0.02%) to 94.763% when pulling 1a11586 on sadikovi:update-documentation into 1fd277a on sunchao:master.

sunchao

Thanks @sadikovi . This is excellent work!

sunchao · 2018-04-10T16:58:57Z

README.md

+}
+```
+See crate documentation on available API.
+
 ## Requirements
 - Rust nightly
 - Thrift 0.11.0 or higher


Not quite related, but we may want to also point out the parquet format version that we support - currently it is 2.3.2?

sunchao · 2018-04-11T03:31:24Z

src/basic.rs

+  /// - FLOAT - 4 bytes per value, stored as little-endian.
+  /// - DOUBLE - 8 bytes per value, stored as little-endian.
+  /// - BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes.
+  /// FIXED_LEN_BYTE_ARRAY - just the bytes are stored.


nit: missed - at the beginning.

sunchao · 2018-04-11T03:48:05Z

src/lib.rs

 #[macro_use]
 pub mod errors;
 pub mod basic;
 pub mod data_type;
 mod parquet_thrift;

 #[macro_use]
-pub mod util;


This will break cargo bench. We need to find a way to expose only the necessary APIs.

sunchao · 2018-04-11T03:49:38Z

src/schema/types.rs

-/// Conversion from Thrift equivalents
-
+/// Conversion from Thrift equivalents.
+/// Should be considered private.


Why pub can't be removed? add a TODO?

sadikovi · 2018-04-11T10:06:47Z

I just need to document encoding and decoding modules, and review again.

sadikovi · 2018-04-12T21:54:51Z

@sunchao I think it is ready for review now. Would you mind having a look again? Thanks!

sunchao

LGTM! with a few minor comments.

sunchao · 2018-04-13T04:04:10Z

README.md

+```
+See crate documentation on available API.
+
+## Versions


instead of Versions, maybe Supported Parquet Version?
Also, the version we support right now is 2.4.0

sunchao · 2018-04-13T04:32:26Z

src/schema/types.rs

-  // Creates a new `PrimitiveType` instance from the gathered attributes.
-  // This also checks various illegal conditions and returns `Err` if that that happen.
+  /// Creates a new `PrimitiveType` instance from the gathered attributes.
+  /// Also checks various illegal conditions and returns `Err` if that that happen.


nit: extra that. happen -> happens.

sadikovi · 2018-04-13T09:37:02Z

I updated the code, can you have a look again? Thanks!

sunchao

LGTM!

sadikovi · 2018-04-13T22:45:20Z

Thanks!

sadikovi and others added 18 commits April 5, 2018 20:19

update public modules

eb6dd87

add docs

331fe7d

add doc build in readme

cc88b2f

update lib doc

5801b6e

update docs for basic.rs

b3cc916

add license header

efc7c37

add docs for basic and errors

ecc1bc9

add doc for data_type

9d3cee0

add schema doc

9da396e

update schema docs

0092fca

add record docs

1c2420e

add column docs

f37efd8

add file doc and some minor updates

a7f3bb7

update readme

c2e1bdd

add docs.rs link

99a1353

update docs

5d34f72

update file docs

db3ae7a

add comment for column/reader.rs tests

08fd9e8

sunchao reviewed Apr 11, 2018

View reviewed changes

sadikovi added 6 commits April 11, 2018 20:34

add parquet version, minor updates

655d5b6

update comment for types.rs

c6f350c

reexport modules

54a1c0d

add compression docs

c130834

update docs

e6c9803

add memory docs

2e59245

sadikovi added 2 commits April 12, 2018 10:19

update global doc

5bc4cd6

add decoding docs

d566524

sadikovi added 4 commits April 12, 2018 21:30

update docs

b96050b

update docs

605a1fa

fix doc tests

c78f550

update from_thrift comments

af13421

sunchao approved these changes Apr 13, 2018

View reviewed changes

sadikovi added 2 commits April 13, 2018 20:28

update parquet version in readme

51a4965

update comments

1a11586

sunchao approved these changes Apr 13, 2018

View reviewed changes

sunchao merged commit 5e4758c into sunchao:master Apr 13, 2018

sadikovi deleted the update-documentation branch April 13, 2018 22:45

sunchao mentioned this pull request Apr 14, 2018

Releasing 0.2.0? #88

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update documentation #81

Update documentation #81

sadikovi commented Apr 10, 2018

sadikovi commented Apr 10, 2018

coveralls commented Apr 10, 2018 •

edited

Loading

sunchao left a comment

sunchao Apr 10, 2018

sunchao Apr 11, 2018

sunchao Apr 11, 2018

sunchao Apr 11, 2018

sadikovi commented Apr 11, 2018

sadikovi commented Apr 12, 2018

sunchao left a comment

sunchao Apr 13, 2018

sunchao Apr 13, 2018

sadikovi commented Apr 13, 2018

sunchao left a comment

sadikovi commented Apr 13, 2018

Update documentation #81

Update documentation #81

Conversation

sadikovi commented Apr 10, 2018

sadikovi commented Apr 10, 2018

Publishing docs

coveralls commented Apr 10, 2018 • edited Loading

sunchao left a comment

Choose a reason for hiding this comment

sunchao Apr 10, 2018

Choose a reason for hiding this comment

sunchao Apr 11, 2018

Choose a reason for hiding this comment

sunchao Apr 11, 2018

Choose a reason for hiding this comment

sunchao Apr 11, 2018

Choose a reason for hiding this comment

sadikovi commented Apr 11, 2018

sadikovi commented Apr 12, 2018

sunchao left a comment

Choose a reason for hiding this comment

sunchao Apr 13, 2018

Choose a reason for hiding this comment

sunchao Apr 13, 2018

Choose a reason for hiding this comment

sadikovi commented Apr 13, 2018

sunchao left a comment

Choose a reason for hiding this comment

sadikovi commented Apr 13, 2018

coveralls commented Apr 10, 2018 •

edited

Loading