Add a unit test to test repeated updates of resource-group. #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/pulsarbot run-failure-checks |
1 similar comment
|
/pulsarbot run-failure-checks |
…lsar processes (apache#11138) ### Motivation - allows Netty to use reflection access on JDK9+ - fixes `jvm_memory_direct_bytes_used` Prometheus metric for Bookkeeper (Bookie). The direct memory usage metrics are disabled in Bookie on JDK11 unless `-Dio.netty.tryReflectionSetAccessible=true` is used. - enables Netty to use more efficient byte buffer implementation such as UnpooledUnsafeNoCleanerDirectByteBuf in JDK9+ (JDK 11 in this case). This is already used in JDK8. In JDK9+, it is necessary to set `-Dio.netty.tryReflectionSetAccessible=true` to use UnpooledUnsafeNoCleanerDirectByteBuf. This is necessary for preventing performance regressions in this area. ### Modifications - pass `-Dio.netty.tryReflectionSetAccessible=true` JVM option by default to all pulsar process started with `bin/pulsar` script.
### Motivation Use Awaitility to instead of Thread sleep ### Modifications - Use Awaitility to instead of Thread sleep
- Fixed some useless loops. - Fixed incorrect usage of Array tostring method.
…amicConfigurationBoolean. (apache#11171) ### Motivation 1. Adjust the implementation of getDynamicConfigurationDouble and DynamicConfigurationBoolean. 2. Fix typos.
* fix: python client docs missing pkg import Signed-off-by: Eric Shen <ericshenyuhao@outlook.com> * fix: apply to previous version Signed-off-by: Eric Shen <ericshenyuhao@outlook.com>
…1160) Currently, website construction test is not very convenient, you need to merge branches to see the effect, in order to facilitate local testing, I added this tool script, after building the website, you can execute `sh debug-site.sh` locally, you can access the latest build locally Website
…f cli tool (apache#11169) ### Master Issue: apache#10040 ### Motivation Support auto generate HTML page for pulsar client cli tool, for example: https://github.com/apache/pulsar/tree/asf-site/content/tools/pulsar-admin ### Modifications - Add auto script for pulsar-perf - Add drop-menu for pulsar-perf
### Motivation Some user networks may have a layer of socks5 proxy, so the client side is added socks5 proxy handler to support socks5 protocal. ### Modifications Add Socks5ProxyHandler in ConnectionPool as the first handler for producer and consumer.
Co-authored-by: wangjialing <wangjialing@cmss.chinamobile.com>
The hard coded token has expired causing the test to fail. Instead of hardcoding the token, we can generate the token dynamically and avoid this problem in future Co-authored-by: Surinder Singh <surinders@splunk.com>
…nc (apache#11183) The issue will happens after satisfying the following conditions: 1. The messages are added to the incoming queue before reading messages 2. The result future of the readNextAsync been complete before call future.whenComplete by users, This won't always appear. After that, since we are using the IO thread to call the callback of the hasMessageAvailableAsync, so the IO thread will process the message.getValue(). Then might get a deadlock as followings: ``` java.util.concurrent.CompletableFuture.get() CompletableFuture.java:1998 org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.getSchemaInfoByVersion(byte[]) AbstractMultiVersionReader.java:115 org.apache.pulsar.client.impl.schema.reader.MultiVersionAvroReader.loadReader(BytesSchemaVersion) MultiVersionAvroReader.java:47 org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader$1.load(BytesSchemaVersion) AbstractMultiVersionReader.java:52 org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader$1.load(Object) AbstractMultiVersionReader.java:49 com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(Object, CacheLoader) LocalCache.java:3529 com.google.common.cache.LocalCache$Segment.loadSync(Object, int, LocalCache$LoadingValueReference, CacheLoader) LocalCache.java:2278 com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(Object, int, CacheLoader) LocalCache.java:2155 com.google.common.cache.LocalCache$Segment.get(Object, int, CacheLoader) LocalCache.java:2045 com.google.common.cache.LocalCache.get(Object, CacheLoader) LocalCache.java:3951 com.google.common.cache.LocalCache.getOrLoad(Object) LocalCache.java:3974 com.google.common.cache.LocalCache$LocalLoadingCache.get(Object) LocalCache.java:4935 org.apache.pulsar.client.impl.schema.reader.AbstractMultiVersionReader.read(byte[], byte[]) AbstractMultiVersionReader.java:86 org.apache.pulsar.client.impl.schema.AbstractStructSchema.decode(byte[], byte[]) AbstractStructSchema.java:60 org.apache.pulsar.client.impl.MessageImpl.getValue() MessageImpl.java:301 org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.refreshTopicPoliciesCache(Message) SystemTopicBasedTopicPoliciesService.java:302 org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$initPolicesCache$7(SystemTopicClient$Reader, Throwable, CompletableFuture, Message, Throwable) SystemTopicBasedTopicPoliciesService.java:254 org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda$817.accept(Object, Object) java.util.concurrent.CompletableFuture.uniWhenComplete(Object, BiConsumer, CompletableFuture$UniWhenComplete) CompletableFuture.java:859 java.util.concurrent.CompletableFuture.uniWhenCompleteStage(Executor, BiConsumer) CompletableFuture.java:883 java.util.concurrent.CompletableFuture.whenComplete(BiConsumer) CompletableFuture.java:2251 org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.lambda$initPolicesCache$10(SystemTopicClient$Reader, CompletableFuture, Boolean, Throwable) SystemTopicBasedTopicPoliciesService.java:246 org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService$$Lambda$725.accept(Object, Object) org.apache.pulsar.client.impl.ClientCnx.handleGetLastMessageIdSuccess(PulsarApi$CommandGetLastMessageIdResponse) ClientCnx.java:468 ``` Since we are introduced the internal thread pool for handling the client internal executions. So the fix is using the internal thread to process the callback of the hasMessageAvailableAsync
Co-authored-by: kimura <takeshki@yahoo-corp.jp>
…ache#11194) * Fixed possible deadlock in the initialization of MLTransactionLog * Fixed tests
* PIP-45: Migrate OwnershipCache to use MetadataStore * Import checkstyle * Fixed typo * Addressed comment * Fixed test * Test fixes * One more test fix * Fixed possible deadlock in the initialization of MLTransactionLog * Fixed tests
apache#11166) * Add a call ACK / unack example to avoid confusing the retrial mechanism for beginners Co-authored-by: yangde <yangde@myai.tech>
### Motivation Currently, the ledger rollover scheduled task will execute before reach the ledger maximum rollover time, this will cause the ledger doesn't roll over in time. ### Modifications Only make the ledger rollover scheduled task after the ledger created successfully. If the scheduled task was executed when there is no entry in the current ledger, the scheduled task will not be re-executed, and if there is new entry is added the ledger will rollover. ### Verifying this change Add a unit test to verify the ledger could be rolled over in time.
…he#11056) ### Motivation Fixes apache#8668 ### Modifications Expose `PulsarClient` via `BaseContext`, and allow connectors to use the inherited pulsar client from function worker to produce/consume messages. ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change is already covered by existing tests, such as: - PulsarOffsetBackingStoreTest - KafkaConnectSourceTest - KafkaConnectSinkTest ### Does this pull request potentially affect one of the following parts: - The public API: `SourceContext` and `SinkContext` need to implement the `getPulsarClient` method
Signed-off-by: Eric Shen <ericshenyuhao@outlook.com>
…#11210) * PIP-45: Converted bookies REST endpoint to use metadata store * Removed unused imports
* [PIP-88] Replicate schemas accross clusters Here is the proposal: https://github.com/apache/pulsar/wiki/PIP-88%3A-Replicate-schemas-across-multiple For the implementation, we just need to set the correct SchemaInfo for the replicated message and using the AutoProduceByte schema for the producer of the
…pache#11557) Fixes apache#11551 ### Motivation Currently there're some bugs of C++ client and some tests cannot pass: 1. Introduced from apache#10601 because it changed the behavior of the admin API to get partition metadata while the C++ implementation relies on the original behavior to create topics automatically. So any test that uses HTTP lookup will fail. - AuthPluginTest.testTlsDetectHttps - AuthPluginToken.testTokenWithHttpUrl - BasicEndToEndTest.testHandlerReconnectionLogic - BasicEndToEndTest.testV2TopicHttp - ClientDeduplicationTest.testProducerDeduplication 2. Introduced from apache#11029 and apache#11486 , the implementation will iterate more than once even there's only one valid resolved IP address. - ClientTest.testConnectTimeout In addition, there's an existed flaky test from very early time: ClientTest.testLookupThrottling. Python tests are also broken. Because it must run after all C++ tests passed, they're also not exposed. 1. Some tests in `pulsar_test.py` might encounter `Timeout` error when creating producers or consumers. 2. Some tests in `schema_test.py` failed because some comparisons between two `ComplexRecord`s failed. Since the CI test of C++ client would never fail after apache#10309 (will be fixed by apache#11575), all PRs about C++ or Python client are not verified even if CI passed. Before apache#11575 is merged, we need to fix all existed bugs of C++ client. ### Modifications Corresponding to the above tests group, this PR adds following modifications: 1. Add the `?checkAllowAutoCreation=true` URL suffix to allow HTTP lookup to create topics automatically. 2. When iterating through a resolved IP list, increase the iterator first, then run the connection timer and try to connect the next IP. Regarding to the flaky `testLookupThrottling`, this PR adds a `client.close()` at the end of test and fix the `ClientImpl::close` implementation. Before this PR, if there're no producers or consumers in a client, the `close()` method wouldn't call `shutdown()` to close connection poll and executors. Only after the `Client` instance was destructed would the `shutdown()` method be called. In this case, this PR calls `handleClose` instead of invoking callback directly. In addition, change the log level of this test to debug. This PR also fixes the failed timeout Python tests, some are caused by incorrect import of classes, some are caused by `client` was not closed. Regarding to Python schema tests, in Python2, `self.__ne__(other)` is not equivalent to `not self.__eq__(other)` when the default `__eq__` implementation is overwritten. If a `Record` object has a field whose type is also `Record`, the `Record.__ne__` method will be called, see https://github.com/apache/pulsar/blob/ddb5fb0e062c2fe0967efce2a443a31f9cd12c07/pulsar-client-cpp/python/pulsar/schema/definition.py#L138-L139 but it just uses the default implementation to check whether they're not equal. The custom `__eq__` method won't be called. Therefore, this PR implement `Record.__ne__` explicitly to call `Record.__eq__` so that the comparison will work for Python2. ### Verifying this change We can only check the workflow output to verify this change.
…e#11119) Fixes #apache#10495 ### Motivation MsgDropRate info is missing after NonPersistentTopics admin api merged with Topics admin api. This PR is trying to fix this. ### Modifications Seems due to API merging, data is not properly deserialized in admin client. And also due to the added TopicsStats interface, the field hiding causing weird behavior with Jackson so fields in NonPersistentTopicStatsImpl intended to hide superclass' fields are not shown in output. Fixing by not using same field name to hide superclass fields and use @JsonIgnore to hide them from output. And add new fields to store subscription/publisher/replicator info for NonPersistentTopic. This does change the output name of those info, but it only changed in cli output, for admin client the old getSubscriptions/getSubscriptions/getReplication will still work.
### Motivation - fixes issue that cpp build doesn't fail when tests fail - merge after apache#11557 ### Additional context - apache#11557 (comment) - https://github.com/apache/pulsar/pull/10309/files#r683626563 ### Modifications - `set -o pipefail;` is required when using `| cat`
Fix delete sub then delete pending ack managedledger.
* Update helm-overview.md * Update helm-overview.md * Update helm-prepare.md * Update site2/docs/helm-overview.md Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com> * Update site2/docs/helm-overview.md Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com> * Update helm-prepare.md * Update helm-install.md * Update helm-deploy.md * Update helm-deploy.md * Update helm-deploy.md * Update helm-deploy.md Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com>
* Update client-libraries-python.md * Update site2/docs/client-libraries-python.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> * Update client-libraries-python.md * Update client-libraries-python.md * Update site2/docs/client-libraries-python.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com>
…1600) ### Motivation Add timeout for the reader.hasMessageAvailable(hasMsg). ### Modifications Use `sendRequestWithId` instead if `sendCommand`, and former will get `OperationTimeout`.
…/nio/ByteBuffer when enabling topic metadata compression (apache#11594) ### Motivation Related to apache#11593. This PR is copying the bytes to the DirectBuf to get around issue apache#11593. The created issue airlift/aircompressor#133 for tracking fix from [aircompressor](https://github.com/airlift/aircompressor) ### Verifying this change Make the `ManagedLedgerCompressionTest` running on JDK8
### Motivation Make `StatsPeroidSeconds` configurable. ### Modifications - Move ‘StatsPeriodSeconds’ from ManagedLedgerFactoryImpl to ManagedLedgerFactoryConfig. - Add config `managedLedgerStatsPeriodSeconds`.
…#11595) Fixes apache#11533 ### Motivation Refer to issue apache#11533 , currently, if users redefined the same `Record` or `Enum` in `Record`, the schema info isn't reused the defined name, this does not match the Avro schema info format. ### Modifications Add a new method `schema_info(self, defined_names)` in `Record`, `Array`, `Map`, and `Enum`, all defined names will be added in the parameter `defined_names` when users use a defined `Record`, or `Enum`, the schema info will use the name of the defined `Record` or `Enum` as the type.
* update helm chart link * Update helm-overview.md unify the written form of `Apache Pulsar Helm chart`.
…ic or sub have been deleted. (apache#11304) ## Motivation now, when topic have been deleted we will not handle the `endTxnOnTopic` and endTxnOnSub. now, when sub have been deleted we will not handle the `endTxnOnSub`. when topic or sub have been deleted, we should return success to tc on `endTxnOnTopic` and endTxnOnSub operation. ## implement when topic not exist in this broker, we should judge the topic if it have been created. if not created, we should return success.
…e#11589) * Updated Python installation steps for Mac * Update site2/docs/client-libraries-python.md Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com> Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>
* Update deploy-bare-metal.md * Update deploy-bare-metal.md * Create deploy-bare-metal.md * Update deploy-bare-metal.md * Update deploy-bare-metal.md * Update site2/docs/deploy-bare-metal.md Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com> * Update site2/docs/deploy-bare-metal.md Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com> * Update deploy-bare-metal.md * Update site2/docs/deploy-bare-metal.md Co-authored-by: sijia-w <53718687+sijia-w@users.noreply.github.com> * Update site2/docs/deploy-bare-metal.md Co-authored-by: sijia-w <53718687+sijia-w@users.noreply.github.com> * Update deploy-bare-metal.md pulsar->Pulsar * Update site2/docs/deploy-bare-metal.md Co-authored-by: sijia-w <53718687+sijia-w@users.noreply.github.com> * Update site2/docs/deploy-bare-metal.md Co-authored-by: sijia-w <53718687+sijia-w@users.noreply.github.com> * Update site2/docs/deploy-bare-metal.md Co-authored-by: sijia-w <53718687+sijia-w@users.noreply.github.com> * Create deploy-bare-metal.md * Update site2/docs/deploy-bare-metal.md * Update site2/docs/deploy-bare-metal.md Co-authored-by: Jennifer Huang <47805623+Jennifer88huang@users.noreply.github.com> Co-authored-by: sijia-w <53718687+sijia-w@users.noreply.github.com>
…apache#11611) Co-authored-by: Jerry Peng <jerryp@splunk.com>
* Fix schema-understand doc. * Add some version change Co-authored-by: congbo <congbobo184@github.com>
### Motivation Currently, there is no visibility about the following activities: - How many bytes are written from a cursor update? - How many bytes are read from loading a cursor? So when the bookkeeper cluster is having heavy traffic, it is hard to tell which topic or namespace contributes most of the traffic. Add metrics at the broker about how many bytes are written and read per cursor/namespace. ### Modifications Add metrics `writeLedgerSize`, `writeLedgerLogicalSize`, `readLedgerSize`.
## Modivation Fix the Issue of apache#11481 In standalone mode, pulsar 2.8.0 cannot be used normally when the transaction is started ## CauseBy ```getTopic```was executed twice when FunctionWorkService . ```getTopicIfExists```make ```createIfMissing = false``` When the execution ends. ```PersistentSubscription``` will create a ledger for the subscription when transaction was turned on. ```new MetadataNotFoundException("Managed ledger not found")```was thrown when calling ```MetaStoreImpl::getManagedLedgerInfo``` ## implement Create a separate ManagerLedgerConfig for PendingAck ## verify Add testSubscriptionRecreateTopic in TransactionTest
bharanic-dev
force-pushed
the
metadata-missing-events
branch
from
e518245 to
7ac24c0
Compare
bharanic-dev
force-pushed
the
metadata-missing-events
branch
from
7ac24c0 to
2daf479
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(If this PR fixes a github issue, please add
Fixes #<xyz>.)Fixes #
(or if this PR is one task of a github issue, please add
Master Issue: #<xyz>to link to the master issue.)Master Issue: #
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation