Skip to content
This repository has been archived by the owner on Jan 24, 2024. It is now read-only.

[FEATURE] support different compress type #78

Open
jiazhai opened this issue Jan 21, 2020 · 0 comments
Open

[FEATURE] support different compress type #78

jiazhai opened this issue Jan 21, 2020 · 0 comments
Labels
triage/week-15 type/feature Indicates new functionality workflow::triaged

Comments

@jiazhai
Copy link
Contributor

jiazhai commented Jan 21, 2020

currently we did not do compress while produce and fetch messages. It would be good to add support for this feature.
The message handling are mostly in class "MessageRecordUtils".
@jiazhai jiazhai added the type/feature Indicates new functionality label Jan 21, 2020
michaeljmarshall pushed a commit to michaeljmarshall/kop that referenced this issue Apr 24, 2023
@eolivelli
[transactions] Implement KIP-664 DescribeProducers (streamnative#78)
c3376e5
gaoran10 pushed a commit to gaoran10/kop that referenced this issue Jul 30, 2023
@eolivelli @gaoran10
[transactions] Implement KIP-664 DescribeProducers (streamnative#78)
d60d786 
(cherry picked from commit c3376e5)
Demogorgon314 pushed a commit to Demogorgon314/kop that referenced this issue Oct 30, 2023
@BewareMyPower
Try fixing flaky testListOffsetForEmptyRolloverLedger and add more lo…
b3e3e0b 
…gs (streamnative#78)

Master issue: streamnative/ksn#26

### Motivation

`testListOffsetForEmptyRolloverLedger` is very flaky, from the logs we
can see the ledger rollover did not clear the last ledger:

```
15:32:07.523 [awaitility-thread:io.streamnative.pulsar.handlers.kop.e2e.MultiLedgerTest@227] INFO  io.streamnative.pulsar.handlers.kop.e2e.MultiLedgerTest - Managed ledger status: [LedgerOpened], ledgers info: [{9=ledgerId: 9
entries: 2
size: 116
timestamp: 1691508727345
, 10=ledgerId: 10
timestamp: 0
}]
...
15:32:22.256 [awaitility-thread:io.streamnative.pulsar.handlers.kop.e2e.MultiLedgerTest@227] INFO  io.streamnative.pulsar.handlers.kop.e2e.MultiLedgerTest - Managed ledger status: [LedgerOpened], ledgers info: [{9=ledgerId: 9
entries: 2
size: 116
timestamp: 1691508727345
, 10=ledgerId: 10
timestamp: 0
}]
```

It appears that the `rollCurrentLedgerIfFull` does not work as expected.
The condition check is:

```java
if (currentLedgerEntries > 0 && currentLedgerIsFull()
    && STATE_UPDATER.compareAndSet(this, State.LedgerOpened, State.ClosingLedger)) {
```

`currentLedgerEntries` and `state` should be fine, the only issue is
that `currentLedgerIsFull` could return false when metadata service is
not available:

```java
    private boolean currentLedgerIsFull() {
        if (!factory.isMetadataServiceAvailable()) {
            // We don't want to trigger metadata operations if we already know that we're currently disconnected
            return false;
        }
```

### Modifications

- Retry `rollCurrentLedgerIfFull` when there are more than 1 ledgers
- Wait until `isMetadataServiceAvailable()` returns true.
- Set `minimumRolloverTimeMs` with 0 so that `currentLedgerIsFull()`
returns true once `currentLedgerEntries >=
config.getMaxEntriesPerLedger()` is satisfied.
- Enable debug logs for `org.apache.bookkeeper.mledger.impl` so that
`ManagedLedgerImpl` could have debug logs.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
triage/week-15 type/feature Indicates new functionality workflow::triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sijie @jiazhai