Skip to content

Commit

Permalink
Move test generation code into README (deephaven#4965)
Browse files Browse the repository at this point in the history
This (hopefully) fixes a problem where GH security is alerting that we are using an insecure version of pyarrow (<14.0.1).
  • Loading branch information
devinrsmith authored Dec 19, 2023
1 parent 7e22770 commit 4d5d618
Show file tree
Hide file tree
Showing 7 changed files with 66 additions and 90 deletions.
66 changes: 66 additions & 0 deletions extensions/parquet/table/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# extensions-parquet-table

## Test data

Some of the test data under [src/test/resources](src/test/resources) was generated with the following snippet:

```python
import pandas as pd
import numpy as np

df = pd.DataFrame(
{
"a": list("abc"),
"b": list(range(1, 4)),
"c": np.arange(3, 6).astype("u1"),
"d": np.arange(4.0, 7.0, dtype="float64"),
"e": [True, False, True],
"f": pd.date_range("20130101", periods=3),
"g": pd.date_range("20130101", periods=3, tz="US/Eastern"),
"h": pd.Categorical(list("abc")),
"i": pd.Categorical(list("abc"), ordered=True),
}
)

df.to_parquet("uncompressed.parquet", compression=None)
df.to_parquet("brotli.parquet", compression="brotli")
df.to_parquet("gzip.parquet", compression="gzip")
df.to_parquet("lz4.parquet", compression="lz4")
df.to_parquet("snappy.parquet", compression="snappy")
df.to_parquet("zstd.parquet", compression="zstd")
```

Using the following requirements:

```requirements
# for src/test/resources/e0
numpy==1.24.2
pandas==1.5.3
pyarrow==5.0.0
python-dateutil==2.8.2
pytz==2022.7.1
six==1.16.0
```

```requirements
# for src/test/resources/e1
numpy==1.24.2
pandas==1.5.3
pyarrow==11.0.0
python-dateutil==2.8.2
pytz==2022.7.1
six==1.16.0
```

```requirements
# for src/test/resources/e2
cramjam==2.6.2
fastparquet==2023.2.0
fsspec==2023.3.0
numpy==1.24.2
packaging==23.0
pandas==1.5.3
python-dateutil==2.8.2
pytz==2022.7.1
six==1.16.0
```
23 changes: 0 additions & 23 deletions extensions/parquet/table/src/test/e0.py

This file was deleted.

6 changes: 0 additions & 6 deletions extensions/parquet/table/src/test/e0.requirements.txt

This file was deleted.

23 changes: 0 additions & 23 deletions extensions/parquet/table/src/test/e1.py

This file was deleted.

6 changes: 0 additions & 6 deletions extensions/parquet/table/src/test/e1.requirements.txt

This file was deleted.

23 changes: 0 additions & 23 deletions extensions/parquet/table/src/test/e2.py

This file was deleted.

9 changes: 0 additions & 9 deletions extensions/parquet/table/src/test/e2.requirements.txt

This file was deleted.

0 comments on commit 4d5d618

Please sign in to comment.