Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distributed Log based Pulsar #1

Closed
wants to merge 37 commits into from
Closed

Distributed Log based Pulsar #1

wants to merge 37 commits into from

Conversation

ArvinDevel
Copy link

Motivation

enable DLog as Pulsar's storage

Modifications

ManagedLedger relative mostly

Result

the storage manage of Pulsar will change

@sijie
Copy link
Owner

sijie commented Oct 12, 2017

@ArvinDevel I missed this pull request. I will take a look at the weekend.

@sijie
Copy link
Owner

sijie commented Oct 26, 2017

@ArvinDevel it seem this pull request was based on very late master. I've rebased my master to lastest apache master. can you update this pull request, so it is easier for me to review?

merlimat and others added 28 commits October 26, 2017 14:02
@merlimat
Allow to configure the default number of bundles for new namespaces (a…
8448198 
…pache#854)
@rdhabalia @merlimat
Introduce auto bundle split and unloading of split bundle in ModularL…
eb9e5fc 
…oadManager (apache#857)
@rdhabalia
Fix: compilation error for NamespaceServiceTest (apache#863)
6bd024d
@rdhabalia @merlimat
Make spark consumer-listener lambda Serializable (apache#866)
3e965b2
@rdhabalia @merlimat
Fix: profobuf interface generation on doc (apache#864)
8ba2a7c
@merlimat
Pulsar-testclient should not fetch the shaded pulsar-client dependency (
e12d0fd 
apache#868)
@afalko @merlimat
Add proxy configuration for kubernetes (apache#856)
5c58367
@ArvinDevel
add dlog impl
3428cc6
@ArvinDevel
Initialize DlogBasedPulsar
3c2e17a
@ArvinDevel
sub update
ab32767
@ArvinDevel
update impl
07ef7f2
@ArvinDevel
9-6
8f5019f
@ArvinDevel
update before rebase
c3daa0a
@ArvinDevel
clear before commit
bb27f58
@ArvinDevel
internal update
8e1f778
@ArvinDevel
update dlog impl
9836883
@ArvinDevel
fix test error
8c58d8d
@ArvinDevel
"enable unit test for DlogBasedManagedLedger"
af78365
@ArvinDevel
"promote the impl, fix null pointer error"
1a7263a
@ArvinDevel
"change mangedCursor's bk to dlog's bk(cut managedLedgerConfig's comp…
c200831 
…atibility with original)"
@ArvinDevel
"update managedLedger test "
6d89b57
@ArvinDevel
"fix some error, such as change ManagedLedgerConfig to DlogBased..., …
0f2d8c0 
…remove mock bookie when setup"
@ArvinDevel
"fix some error, enable cache"
1721a45
@ArvinDevel
"bump netty to 4.1, move log reader to cache"
2e19339
@ArvinDevel
"refactor initialize log writer"
e1d43cd
@ArvinDevel
"unit test use bkshade-jar"
8883157
@ArvinDevel
"disable numEntriesPerLedger test, because dlog doesn't has correspon…
2979c13 
…ding method, and this setting is not necessary enough"
@ArvinDevel
"set read timeout, disable some tests relative to MaxEntriesPerLedger"
de42a7c
sijie added a commit that referenced this pull request Mar 4, 2018
@sijie
Create pulsar-functions module (#1)
72092c4 
* Create pulsar-functions module

* rename `sdk` package to `api`
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Instance (#3)
a13d5ce 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added a simple Instance to run the exposed user interfaces

* Addressed pr comments plus added tests

* Remove unused import
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Spawner (#11)
e30808c 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added the first cut of the Spawner

* Added more func

* Use already existing FunctionConfig

* ompiling version

* Rationalized the use of data structures

* Completed Spawner

* Fixed the unittest
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Bugcix (#12)
5e76557 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Make Spawner class public, Use Lombok for getter/setter
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Add examples (apache#14)
bb6409f 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added a simple example
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Fix bugs (apache#15)
da4957b 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* 1. Start only when there is a subscriber
2. subscription is shared
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Minor stuff (apache#17)
5b0ed1b 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added join interface to Spawner.
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Refactor (apache#18)
52c9c24 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Refactored serialization/deserialization into an interface
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Simplestringserde (apache#19)
85e1882 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added a simple String based serde
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Opt (apache#22)
9766676 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Don't wait for the result. Instead chain it via CompletedFuture
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Cleanup (apache#23)
fc1f71c 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Cleaned up the logic of supported types
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Fix tests (apache#24)
5f3b93e 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Fix void type handling
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Buffer limit (apache#25)
f883f16 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added a way to limit the number of outstanding tuples read but still not executed
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
System exception (apache#26)
117f816 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Introduced the concept of systemException to differentiate from userException
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Refactor cli (apache#27)
f026ef8 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Refactor cli to establish a base FunctionCommand interface that all subcommands can piggyback
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Nomenclature (apache#32)
4dd4089 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Made our function hierarchy match pulsar's

* Replace cluster with tenant
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Added Submit Function capability to the pulsar-functions cli (apache#33)
48d4aa4 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Added the first cut of the pulsar submit

* Working
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Fix compilation (apache#35)
da66b2d 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Fix compilation issue
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Asyncack (apache#37)
7fbe342 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Made ack async
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Nonthreadpool (apache#38)
e90b308 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* In case of Kstream style running, we dont need to spawn a executor service to run the message

* Update comment
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Reconcile (apache#40)
4b4bea7 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Reconciled the interface between worker and client with Jerry

* Fixed the update method
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Made Update a put request (apache#41)
a1b1ea5 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Adhere to rest semantics
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Completecli (apache#43)
5077d71 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Adhere to rest semantics

* Complete the list of functions supported by cli
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Fileinterface (apache#45)
5ea8065 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Adhere to rest semantics

* Convert create function to a post request
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Print (apache#46)
5c4b286 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Adhere to rest semantics

* Print the return values for get/list functions. Also print some statements on success
sijie pushed a commit that referenced this pull request Mar 4, 2018
@srkukarni @sijie
Rename functionstate (apache#52)
59ec630 
* Create pulsar-functions module (#1)

* Create pulsar-functions module

* rename `sdk` package to `api`

* Added the first cut of the Java interface for Pulsar functions (#2)

* Adhere to rest semantics

* Renamed FunctionState to FunctionMetaData
@ArvinDevel ArvinDevel closed this Mar 15, 2018
sijie pushed a commit that referenced this pull request Feb 12, 2021
@rdhabalia
[pulsar-function] fix possible deadlock on broker-function service st…
d654d5e 
…artup (apache#9499)

### Motivation

Standalone pulsar broker and function service can have deadlock which also sometime causes failures in function unit-test case. Pulsar function tries to create a subscription which blocks the zk thread and can be cause of possible deadlock. Below is the thread-dump when the deadlock happens and function service start up fails. 

So, remove blocking call while creating subscription using admin-api.



```

"pulsar-load-manager-139-1" apache#717 prio=5 os_prio=31 tid=0x00007f8a68698000 nid=0x3610f waiting on condition [0x00007000156e9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000792a07020> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
        at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97)
        at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.getDynamicConfigurationFromZK(SimpleLoadManagerImpl.java:391)
        at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.getDynamicConfigurationDouble(SimpleLoadManagerImpl.java:401)
        at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.getLoadBalancerBrokerOverloadedThresholdPercentage(SimpleLoadManagerImpl.java:450)
        at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.generateLoadReportForcefully(SimpleLoadManagerImpl.java:1139)
        - locked <0x000000078e53f2d0> (a java.util.HashSet)
        at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.writeLoadReportOnZookeeper(SimpleLoadManagerImpl.java:1295)
        at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:39)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)



"pulsar-ordered-OrderedExecutor-1-0-EventThread" apache#621 daemon prio=5 os_prio=31 tid=0x00007f89fd8e1800 nid=0x2fe07 waiting on condition [0x000070000f7ce000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000792ecb8b0> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalCreateSubscriptionForNonPartitionedTopic(PersistentTopicsBase.java:2060)
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$69(PersistentTopicsBase.java:2034)
        at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase$$Lambda$676/1709042625.accept(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$5(ZooKeeperCache.java:249)
        at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$154/5998675.processResult(Unknown Source)
        at org.apache.bookkeeper.zookeeper.ZooKeeperClient$15$1.processResult(ZooKeeperClient.java:879)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:583)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)


main" #1 prio=5 os_prio=31 tid=0x00007f8aa0013000 nid=0x1603 waiting on condition [0x0000700004160000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007930efe68> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.apply(AsyncHttpConnector.java:178)
        at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:632)
        at org.glassfish.jersey.client.JerseyInvocation$$Lambda$672/237755480.call(Unknown Source)
        at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:654)
        at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:648)
        at org.glassfish.jersey.client.JerseyInvocation$$Lambda$673/197134325.call(Unknown Source)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
        at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:648)
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:631)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:434)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:318)
        at org.apache.pulsar.client.admin.internal.TopicsImpl.createSubscription(TopicsImpl.java:1143)
        at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:454)
        at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1343)
        at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:671)
        at org.apache.pulsar.io.PulsarFunctionE2ETest.setup(PulsarFunctionE2ETest.java:209)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sijie sijie sijie left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ArvinDevel @sijie @merlimat @rdhabalia @afalko