Add URI Handlers and LoadSpec. Deprecate CompressionUtils #37

drcrallen · 2015-02-18T00:35:52Z

Adds URIDataPuller interface for dealing with URI data. I also have a patch incoming to Druid that implements this update.

Also adds a LoadSpec to help solidify the loading of segment data.

Also move CompressionUtils to java-util via metamx/java-util#22

* Requires druid-io/druid-api#37 * Requires metamx/java-util#20

fjy · 2015-02-24T01:20:43Z

src/main/java/io/druid/segment/loading/DataSegmentPuller.java

   * @throws SegmentLoadingException if there are any errors
   */
+  @Deprecated


why did we deprecate this? Do we want the URI to be the standard way to pull down segments? I think there is simplicity in allowing you to just pass a segment and not having to worry about where that segment is stored.

For the Omni segment loader that should be true. For the puller we were effectively implementing our own non-standard uri. This essentially changes the format for the segment load spec to formal URI compliance

drcrallen · 2015-03-02T22:11:11Z

@fjy Updated this one to address the polymorphic pulling stuff. Update to druid coming soon

fjy · 2015-03-18T16:29:47Z

src/main/java/io/druid/segment/loading/LoadSpec.java

+ * A means of pulling segment files into a destination directory
+ */
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
+public interface LoadSpec


are there any jsonsubtypes here for serde?

Since the LoadSpec and the pullers are intimately tied together, I didn't define any subtypes here, instead opting to register the subtypes whenever the pullers are registered. LoadSpec is passed as a Map<String, Object> until it is finally ready to be consumed. At that point both the LoadSpec subtype and the appropriate puller will be registered.

Also, I didn't want to add a bunch of dependencies into the API jar, which would be required to properly couple the actual LoadSpec impl and the SegmentPuller

xvrl · 2015-03-18T18:19:04Z

can we add more details to the description, saying this also moved the CompressionUtils to java-util and like to the corresponding PR?

drcrallen · 2015-03-18T18:20:22Z

@xvrl : added

xvrl · 2015-03-23T19:39:43Z

src/main/java/io/druid/segment/loading/DataSegmentPullerURI.java

+   * @return The time in ms since 1970-01-01Z when the segment was last modified
+   * @throws SegmentLoadingException on error
+   */
+  public long getLastModified(URI uri) throws IOException;


any reason this is not simply a DateTime?

Modified to generic getVersion

xvrl · 2015-03-23T22:09:54Z

src/main/java/io/druid/segment/loading/LoadSpec.java

+  // Hold interesting data about the results of the segment load
+  public static class LoadSpecResult{
+    private final Long size;
+    public LoadSpecResult(Long size){


does size accept null values?

Missed that one in the great Long purge. Fixed

cheddar · 2015-03-30T18:08:59Z

src/main/java/io/druid/utils/CompressionUtils.java

+   * @param outDir     The destination directory to put the resulting file
+   *
+   * @throws IOException on propogated IO exception, IAE if it cannot determine the proper new name for `pulledFile`
+   */
  public static void gunzip(File pulledFile, File outDir) throws IOException


Why not deprecate this as well?

Because the functionality is slightly different (outDir vs outFile) and there is no alternative for specifying an output directory yet.

Doesn't mean you can't deprecate it. There's technically an alternative...

Allright, I'll put a tag in there

* Also moved most functionality of CompresisonUtils away from here (now in java-util) * Added URIDataPuller interface

cheddar · 2015-03-30T18:37:24Z

LGTM

Add URI Handlers and LoadSpec. Deprecate CompressionUtils

* Requires druid-io/druid-api#37 * Requires metamx/java-util#22 * Moves the puller logic to use a more standard workflow going through java-util helpers instead of re-writing the handlers for each impl * General workflow goes like this: 1) LoadSpec makes sure the correct Puller is called with the correct parameters. 2) The Puller sets up general information like how to make an InputStream, how to find a file name (for .gz files for example), and when to retry. 3) CompressionUtils does most of the heavy lifting when it can

drcrallen force-pushed the dataSegmentURIHandlers branch from 6fc7510 to b46b111 Compare February 18, 2015 00:36

drcrallen mentioned this pull request Feb 18, 2015

Overhaul of SegmentPullers to add consistency and retries apache/druid#1132

Merged

drcrallen force-pushed the dataSegmentURIHandlers branch from b46b111 to cb180a4 Compare February 20, 2015 21:57

drcrallen added a commit to metamx/druid that referenced this pull request Feb 20, 2015

Add URI handling to SegmentPullers

b1cafbc

* Requires druid-io/druid-api#37 * Requires metamx/java-util#20

fjy reviewed Feb 24, 2015
View reviewed changes

drcrallen force-pushed the dataSegmentURIHandlers branch from cb180a4 to b24d2f0 Compare March 2, 2015 22:10

drcrallen changed the title ~~Add URI handlers to DataSegmentPuller~~ Add URI Handlers Mar 2, 2015

drcrallen force-pushed the dataSegmentURIHandlers branch 3 times, most recently from 0accc4e to fd2549e Compare March 9, 2015 17:27

drcrallen force-pushed the dataSegmentURIHandlers branch from fd2549e to b5e478e Compare March 11, 2015 23:24

fjy reviewed Mar 18, 2015
View reviewed changes

drcrallen force-pushed the dataSegmentURIHandlers branch from 1e2d806 to 63c17d4 Compare March 18, 2015 18:18

xvrl reviewed Mar 23, 2015
View reviewed changes

drcrallen force-pushed the dataSegmentURIHandlers branch from 63c17d4 to dcdb624 Compare March 23, 2015 20:10

xvrl reviewed Mar 23, 2015
View reviewed changes

drcrallen force-pushed the dataSegmentURIHandlers branch from dcdb624 to c1ce5f1 Compare March 23, 2015 22:19

drcrallen changed the title ~~Add URI Handlers and LoadSpec~~ Add URI Handlers and LoadSpec. Remove CompressionUtils Mar 26, 2015

drcrallen force-pushed the dataSegmentURIHandlers branch from c1ce5f1 to 27a8e04 Compare March 30, 2015 17:35

drcrallen changed the title ~~Add URI Handlers and LoadSpec. Remove CompressionUtils~~ Add URI Handlers and LoadSpec. Deprecate CompressionUtils Mar 30, 2015

cheddar reviewed Mar 30, 2015
View reviewed changes

Changed data pulling for the DataSegmentPuller and LoadSpec

2eb740e

* Also moved most functionality of CompresisonUtils away from here (now in java-util) * Added URIDataPuller interface

drcrallen force-pushed the dataSegmentURIHandlers branch from 27a8e04 to 2eb740e Compare March 30, 2015 18:29

fjy added a commit that referenced this pull request Mar 30, 2015

Merge pull request #37 from drcrallen/dataSegmentURIHandlers

1b3b335

Add URI Handlers and LoadSpec. Deprecate CompressionUtils

fjy merged commit 1b3b335 into druid-io:master Mar 30, 2015

drcrallen deleted the dataSegmentURIHandlers branch March 30, 2015 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add URI Handlers and LoadSpec. Deprecate CompressionUtils #37

Add URI Handlers and LoadSpec. Deprecate CompressionUtils #37

drcrallen commented Feb 18, 2015

fjy Feb 24, 2015

drcrallen Feb 24, 2015

drcrallen commented Mar 2, 2015

fjy Mar 18, 2015

drcrallen Mar 18, 2015

xvrl commented Mar 18, 2015

drcrallen commented Mar 18, 2015

xvrl Mar 23, 2015

drcrallen Mar 23, 2015

xvrl Mar 23, 2015

drcrallen Mar 23, 2015

cheddar Mar 30, 2015

drcrallen Mar 30, 2015

cheddar Mar 30, 2015

drcrallen Mar 30, 2015

cheddar commented Mar 30, 2015

Add URI Handlers and LoadSpec. Deprecate CompressionUtils #37

Add URI Handlers and LoadSpec. Deprecate CompressionUtils #37

Conversation

drcrallen commented Feb 18, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drcrallen commented Mar 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xvrl commented Mar 18, 2015

drcrallen commented Mar 18, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheddar commented Mar 30, 2015