PIP-15: Pulsar Functions #1314

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Fix void type handling

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Added a way to limit the number of outstanding tuples read but still not executed

* Relocate instance and spawner to be under `runtime` package * fix the pom files and travis build

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Introduced the concept of systemException to differentiate from userException

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Refactor cli to establish a base FunctionCommand interface that all subcommands can piggyback

…JavaInstance (apache#30) ### Motivation SerDe is user defined class which used for serializing and deserializing objects for functions. So the initialization of a SerDe class should happen along with Function class and done by the context class loader. Also SerDe should be a generic-typed interface and the type should be consistent with function type. ### Modifications - make SerDe a generic-typed interface - move SerDe initialization to JavaInstance and use the thread context class loader - verify SerDe type - move reflection related code to a util class `Reflections` ### Result - SerDe is a generic-typed interface - JavaInstance initializes both function and serdes and verifies the types.

### Motivation input type can be different output type. so we need two serdes for each function. ### Modifications - add inputSerde and outputSerde - verify both input and output serde to make sure their types are consistent with function types ### Result user run functions with input/output serdes.

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Made our function hierarchy match pulsar's * Replace cluster with tenant

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Added the first cut of the pulsar submit * Working

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Fix compilation issue

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Made ack async

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * In case of Kstream style running, we dont need to spawn a executor service to run the message * Update comment

* Provide a jmh benchmark to benchmark thread container * Rebased to latest master * revert the scripts

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Reconciled the interface between worker and client with Jerry * Fixed the update method

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Adhere to rest semantics

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Adhere to rest semantics * Complete the list of functions supported by cli

* Introduce FunctionStats to collection function related stats * Format license headers

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Adhere to rest semantics * Convert create function to a post request

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Adhere to rest semantics * Print the return values for get/list functions. Also print some statements on success

* Create pulsar-functions module (#1) * Create pulsar-functions module * rename `sdk` package to `api` * Added the first cut of the Java interface for Pulsar functions (#2) * Adhere to rest semantics * Renamed FunctionState to FunctionMetaData

…ality (apache#53) * Renamed FunctionState stuff to something that reflects their functionality * More removal of state

* Refactor to support at-least-once processing This change is refactoring the relationship between Spawner, FunctionContainer and JavaInstance to better support various processing semantics. - ThreadFunctionContainerFactory is a "singleton" across multiple functions. - Spawner, FunctionContainer and JavaInstance are per function. Spawner is manging the lifecycle of function container, in the function container it will folks a thread for running java instance, because it has to load classpathes correctly. - Remove subscribermanager and move consumer and producer logic into container. that means in future if we are supporting process based and docker based container, we can just take the thread container and run it directly as a separate process or a separate docker process. There is no extra communication between spawner and containers. * Create pulsar client in thread container factory

…e processing (apache#59) * Use LinkedBlockingQueue#put to enqueue items * Revert it back to at-most-once processing * remove ack for void functions

* Added atmost once and atleast once semantics * Added ability to specify processing guarantees in command line

)

1. upload packages to bookkeeper will include a UUID 2. incorrect setName - > setNamespace

* Add worker main * Organize function packages in hierarchy - functions are stored under distributedlog://<zkservers>/pulsar/functions/<tenant>/<namespace>/<function>/<package_name> - initialize dlog namespace at the startup of worker - use zookeeperServers in worker config instead of dlogUri to hide dlog from pulsar configuration. make it consistent with broker configuration. * Addressed comments

…ache#68) * Added FunctionActioner to start/stop functions in seperate thread * Fixed a bug

…found (apache#71) * Added more logging. Also changed getFunctions to return 404 when not found * Fixed logging

…field is set or not. (apache#73)

* More graceful warning when args are missing * Make use of Annotations * Tests work

* add client side class validation * adding and fixing * cleaning up

* adding some log messages * fixing

* Function Logging - use log4j2 to route function related logging * Add logging function and include tenant/namespace in logging file path * Add missing files

* Name our threads * We already set the name of runner thread

* fixing bugs and adding assignments datastructure * using fully qualified name

…e#82)

* If the function is not found, send 404 * Fix unittest

* Fix upload path issue * fix logging

* Added getstatus function * Changed to GetFunctionStatus * Added licence header

…r. Also consolidate all info in worker in FunctionRuntimeInfo (apache#88)

* Move worker module out of runtime module * Fix license header issues

* Made use of LimitsConfig in JavaInstanceConfig * Add uncommited file

because pulsar-client is using patched protobuf, while dlog/bookkeeper/functions uses same proto3. - move cli related classes to a CLI module - use pulsar-client (shaded jar) for runtime - use dlog (unshaded jar) in worker

…to (apache#104) * Migrating data structures to protobuf * fixing bugs * removing unneccessary files * fixing dependency issues * fixing wrong dependency * correct proto files * fixing issues * fix worker bug

* Added Process based container factory * Added java instance log4j * Generate uberjar * Added log directory for process container * Better naming * Corrected java command line * More changes to get things to work * Fixed process args * Temp change that needs to be reverted * Merge conflict * Siwthced to gprc 1.5.0 for netty compat * Getter for ProcessBuilder * Added test * Added container info * Send Internal Server Error if we have trouble with the request * Dont use unsupported apis * Revert changes * Make Thread container as default * Fix build

…pache#106)

…apache#108)

* Provide a shade `pulsar-client-admin` module * avoid shaded pulsar direct depenencies * Shade `pulsar-client-tools-shaded` as well

…haded jars and avoid shading jackson (apache#116)

* fixing response codes and cli * cleaning up * fixing unit tests

…#118)

* Move SerDe to api * Incorporate comments

* Added publish interface to the api * Address comments

* Added ability to specify subscription type * Fixed build * Do not expose subscription type in cmd line

* Added runtime descriptor in functionConfig * Fixed unittest

* Renamed RequestHandler to PulsarFunction * Fixed header

* adding user acking and fixing bugs * cleaning up

apache#135) * First cut of Python Instance that just parses the arguments and starts a server to process incoming requests * Add license header

* adding windowing to Pulsar functions * cleaning up * relocating code

…nce required files into `target/python-instance` (apache#137) * Provide a copy task in runtime package phase to copy all python-instance required files into `target/python-instance` * cp api-python to target/python-instance/pulsar-function fix the licenses headers * Change `pulsar-function` to `pulsarfunction`

* Feature rich python instance * Seperate out python instance from main * Functionally complete python * Fix syntax * Fix errors * More fixes * Semi working version * Fixed more errors in the produce path * User userCodeFile instead of jarFile * ProcessContainer launches correct python process * Stupid identation bug * Downloaded file should have the same name as the module * Added more comments * Fix a log issue * Add licence header

* Retain server instance. * Pass AutoAcking information * By Default set auto acking to true

…he#143)

… world only (apache#144)

* event time windowing for Pulsar Functions * cleaning up * fixing logging

* Overhaul stats in both Java and Python instances * Added licence header

* adding unittest for windowing * cleaning up * removing print line

* fixing spacing * adding end file newline

* changing publish function to accept string for classname * cleaning up * making change for python api

* Remove logic of supported types since serde is now exposed to the user * Fix test

- inject instance id in thread context. so we can have instance id information in function log - rename dlog logging to bk logging, so all bk/dlog logging can be routed there - add more variables to allow overrides by sys variables - fixes some scripts

…tly (apache#169)

1) one with DefaultSerDe 2) one with custom SerDe

* Make Output Serde Optional * Fix unittest

* Infer Missing Arguments * Added unittest

apache#175) * Change the log file name to the name of the function. This greatly assists in the debugging * Fix unittest

apache#176)

* Move SimpleSerDe to api * Renamed SimpleSerDe to DefaultSerDe

…ache#178)

…ltSerDe (apache#179)

…cified (apache#180)

…ache#182) only when needed

* Improve pulsar functions logging - inject instance id in thread context. so we can have instance id information in function log - rename dlog logging to bk logging, so all bk/dlog logging can be routed there - add more variables to allow overrides by sys variables - fixes some scripts * Improve the logic for finding java instance location * Improve configuring log directory for pulsar worker * Fix python location * Fix license header

* Fixing Function get status * cleaning up * adding back comment * removing unneccessary dep

…ache#196) * adding check for failures subroutine and fixing worker delete bug * adding license header

* Correct the order of checking * Fix more logic

…ions (apache#199)

…nsform result to str (apache#208)

* Rename FunctionConfig names to make them more consistent * Added generated python file

* adding checks for parallelism * fixing unit test

* adding instance id to get status * regen python protobuf * cleaning up

…ng version of zookeeper (apache#227)

…ache#229)

…r electing a leader (apache#231)

*Problem* If we assign workerConfig after setting consumer listener, when `ConsumerStats` tries to dump the consumer configuration, it will throw NPE. *Solution* apache#1260 tried to ignore listener fields for json serialization this change focus on avoiding NPE on serializing MembershipManager

* Add `join` interface to container to wait for a container (apache#121) * add producers * Effective-Once support * Fix conflict after rebase * add tests * fix tests * address the test * Add more tests * remove lombok from api * remove getType * Fix concurrent modification issue in the tests * Register send future before enqueuing message * improve the test * Use conditional remove * remove the check

…oint) to WorkerService (apache#239)

…est endpoint) to WorkerService (apache#239)" (apache#241) This reverts commit 8fd1fd1870753ea8fb2576873800380ac2cea3dd.

…point) to WorkerService (apache#242) * Move all the components required for running worker (except rest endpoint) to WorkerService (apache#239) * start worker service

- pass `WorkerService` instance around in servlet context - rename `BaseApiResource` to `FunctionApiResource` - rename `ApiV1Resource` to `FunctionApiV1Resource`

* removing snapshotting * cleaning up

… utils into its own module (apache#248) * Refactored instance related stuff into instance as well as rejiggered utils into its own module * Discard empty files

* Upgrade dlog dependency * Make pulsar functions work after rebase - bump zk version 3.5 - use the same dlog version as bookkeeper - change standalone to use sorted ledger storage manager (this is a temp fix to address the issues after rebase)

…e#255)

- the package will be generated under `dist/target/`

…#262) * Support parallelism parameter while running functions locally * Fixed the generation of java and python files

…#264)

* Move the thread construction to start to avoid illegalstate * Add more checks to see if thread var is init

* Use `distributedlog-core-shaded` in pulsar worker * revert to db ledger storage * Include netty-all * Fix serviceUrl for functions cli

- downloader is writing the package to a temp location - after a package is downloaded, a hardlink is created to link to the temp file. hardlink will ensure one createLink succeed. - after the hardlink is created, the temp files can be deleted. After this change, it allows concurrent downloading without interleaving with each other.

…apache#269) Example: ``` (CONNECTING) /> (CONNECTED) /> ls pulsar/functions/test test-namespace (CONNECTED) /> ls pulsar/functions/test/test-namespace function11 function12 function13 (CONNECTED) /> ls pulsar/functions/test/test-namespace/function11 8e4dcc81-3b00-4bf9-a489-1823b5514788-pulsar-functions-api-examples.jar (CONNECTED) /> ls pulsar/functions/test/test-namespace/function12 0ee8e267-74cb-4178-99ea-6c723298223c-pulsar-functions-api-examples.jar (CONNECTED) /> ls pulsar/functions/test/test-namespace/function13 8f1dea11-0070-434b-ba97-a42b44d8063c-pulsar-functions-api-examples.jar (CONNECTED) /> ```

The `compile` scope in dependency plugin means : compile, provided, and system dependencies

apache#273) * Create ProcessBuilder at during start() so that start() can be called multiple times from Spawner * Fix unittest

…ache#275)

- Remove function-composition

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PIP-15: Pulsar Functions #1314

PIP-15: Pulsar Functions #1314

Commits on Mar 4, 2018