Skip to content

Releases: scrapinghub/exporters

0.6.3

27 Sep 18:15
Compare
Choose a tag to compare

Fix excessive memory consumption in threaded mode

0.6.2

27 Sep 18:08
Compare
Choose a tag to compare

Fix HubStorageReader :

  • Removed deprecated arg for collection scanner.

0.6.1

22 Aug 14:03
Compare
Choose a tag to compare
  • Bugfix when decompressing multi-chunk compressed stream reader
  • Experimental multi-threaded export

0.6.0

15 Aug 13:29
Compare
Choose a tag to compare

New features:

  • Added retry reading streams #326 (partial fix for #322 )
  • Added FlatsonTransform for easy flattening #324

0.5.0

22 Jul 21:18
Compare
Choose a tag to compare

New features:

  • Introduced StreamBasedReader that adds support for custom decompressors and deserializers #316
  • Added support for uncompressed input via NoDecompressor #316
  • Added support for CSV formatted input through CSVDeserializer #316
  • FSReader now ignores directories whose name start with a dot #314
  • Added DupeFilter, a filter that ignores subsequent duplicates based on a key field #321
  • Allow using integer as project id #309

Bug fixes:

  • Support long values in int fields #320

0.4.15

04 Jul 22:44
Compare
Choose a tag to compare
  • Bugfix that prevents the bypass code to swallow exceptions
  • Updates setup.py metadata
  • Adds license info
  • dicttoxml and bz2file are now lazy requirements

0.4.14

29 Jun 12:57
Compare
Choose a tag to compare
  • Bugfix on S3Reader: it was skipping every other key
  • Updates to documentation, added examples dir
  • Mark item as finished in HS reduce writer

v0.1.19

31 Mar 10:38
Compare
Choose a tag to compare

New Features:

  • Azure blob writer
  • Azure file writer
  • Date rages support for filebases
  • Improved logging
  • Added metadata object to modules
  • Optional consistency checks for all file based writers
  • Exported file name is now the same as compressed file name
  • Moved from nose to py test environment
  • Adds integration tests
  • Adds flake8 checks
  • S3 to azure blobs bypass
  • S3 to azure files bypass
  • Possibility of disabling retries

Bug fixes:

  • Fix retries for mail notifications module
  • Fix tests leaving dirty tmp folder
  • Tons and tons of small fixes

0.1.8

08 Mar 23:59
Compare
Choose a tag to compare
  • Fix MailWriter to support loading keys from environment

v0.1.6

07 Mar 16:08
Compare
Choose a tag to compare

New Features:

  • Dropbox writer