Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade bk to 4.16.4 for test #24

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Upgrade bk to 4.16.4 for test #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6a03165
Upgrade bk to 4.16.4 for test
hangc0276 Jan 19, 2024
e764366
Fix failed test
hangc0276 Jan 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 28 additions & 28 deletions distribution/server/src/assemble/LICENSE.bin.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,34 +345,34 @@ The Apache Software License, Version 2.0
- net.java.dev.jna-jna-jpms-5.12.1.jar
- net.java.dev.jna-jna-platform-jpms-5.12.1.jar
* BookKeeper
- org.apache.bookkeeper-bookkeeper-common-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-proto-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-server-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.3.jar
- org.apache.bookkeeper-circe-checksum-4.16.3.jar
- org.apache.bookkeeper-cpu-affinity-4.16.3.jar
- org.apache.bookkeeper-statelib-4.16.3.jar
- org.apache.bookkeeper-stream-storage-api-4.16.3.jar
- org.apache.bookkeeper-stream-storage-common-4.16.3.jar
- org.apache.bookkeeper-stream-storage-java-client-4.16.3.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.16.3.jar
- org.apache.bookkeeper-stream-storage-proto-4.16.3.jar
- org.apache.bookkeeper-stream-storage-server-4.16.3.jar
- org.apache.bookkeeper-stream-storage-service-api-4.16.3.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.16.3.jar
- org.apache.bookkeeper.http-http-server-4.16.3.jar
- org.apache.bookkeeper.http-vertx-http-server-4.16.3.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.3.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.3.jar
- org.apache.distributedlog-distributedlog-common-4.16.3.jar
- org.apache.distributedlog-distributedlog-core-4.16.3-tests.jar
- org.apache.distributedlog-distributedlog-core-4.16.3.jar
- org.apache.distributedlog-distributedlog-protocol-4.16.3.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.3.jar
- org.apache.bookkeeper-native-io-4.16.3.jar
- org.apache.bookkeeper-bookkeeper-common-4.16.4.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.16.4.jar
- org.apache.bookkeeper-bookkeeper-proto-4.16.4.jar
- org.apache.bookkeeper-bookkeeper-server-4.16.4.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.16.4.jar
- org.apache.bookkeeper-circe-checksum-4.16.4.jar
- org.apache.bookkeeper-cpu-affinity-4.16.4.jar
- org.apache.bookkeeper-statelib-4.16.4.jar
- org.apache.bookkeeper-stream-storage-api-4.16.4.jar
- org.apache.bookkeeper-stream-storage-common-4.16.4.jar
- org.apache.bookkeeper-stream-storage-java-client-4.16.4.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.16.4.jar
- org.apache.bookkeeper-stream-storage-proto-4.16.4.jar
- org.apache.bookkeeper-stream-storage-server-4.16.4.jar
- org.apache.bookkeeper-stream-storage-service-api-4.16.4.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.16.4.jar
- org.apache.bookkeeper.http-http-server-4.16.4.jar
- org.apache.bookkeeper.http-vertx-http-server-4.16.4.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.4.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.4.jar
- org.apache.distributedlog-distributedlog-common-4.16.4.jar
- org.apache.distributedlog-distributedlog-core-4.16.4-tests.jar
- org.apache.distributedlog-distributedlog-core-4.16.4.jar
- org.apache.distributedlog-distributedlog-protocol-4.16.4.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.4.jar
- org.apache.bookkeeper-bookkeeper-slogger-api-4.16.4.jar
- org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.4.jar
- org.apache.bookkeeper-native-io-4.16.4.jar
* Apache HTTP Client
- org.apache.httpcomponents-httpclient-4.5.13.jar
- org.apache.httpcomponents-httpcore-4.4.15.jar
Expand Down
6 changes: 3 additions & 3 deletions distribution/shell/src/assemble/LICENSE.bin.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,9 +386,9 @@ The Apache Software License, Version 2.0
- log4j-web-2.18.0.jar

* BookKeeper
- bookkeeper-common-allocator-4.16.3.jar
- cpu-affinity-4.16.3.jar
- circe-checksum-4.16.3.jar
- bookkeeper-common-allocator-4.16.4.jar
- cpu-affinity-4.16.4.jar
- circe-checksum-4.16.4.jar
* AirCompressor
- aircompressor-0.20.jar
* AsyncHttpClient
Expand Down
10 changes: 9 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ flexible messaging model and an intuitive client API.</description>
<!-- apache commons -->
<commons-compress.version>1.21</commons-compress.version>

<bookkeeper.version>4.16.3</bookkeeper.version>
<bookkeeper.version>4.16.4</bookkeeper.version>
<zookeeper.version>3.9.1</zookeeper.version>
<commons-cli.version>1.5.0</commons-cli.version>
<commons-text.version>1.10.0</commons-text.version>
Expand Down Expand Up @@ -2563,5 +2563,13 @@ flexible messaging model and an intuitive client API.</description>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>bk-staging</id>
<name>bk-staging</name>
<url>https://repository.apache.org/content/repositories/orgapachebookkeeper-1091/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
</project>
127 changes: 104 additions & 23 deletions ...ar-metadata/src/test/java/org/apache/bookkeeper/replication/AuditorLedgerCheckerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import lombok.Cleanup;
import org.apache.bookkeeper.bookie.BookieImpl;
Expand Down Expand Up @@ -425,8 +426,16 @@ public void testInnerDelayedAuditOfLostBookies() throws Exception {
// wait for 5 seconds before starting the recovery work when a bookie fails
urLedgerMgr.setLostBookieRecoveryDelay(5);

// shutdown a non auditor bookie; choosing non-auditor to avoid another election
String shutdownBookie = shutDownNonAuditorBookie();
AtomicReference<String> shutdownBookieRef = new AtomicReference<>();
CountDownLatch shutdownLatch = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie = shutDownNonAuditorBookie();
shutdownBookieRef.set(shutdownBookie);
shutdownLatch.countDown();
} catch (Exception ignore) {
}
}).start();

if (LOG.isDebugEnabled()) {
LOG.debug("Waiting for ledgers to be marked as under replicated");
Expand All @@ -442,9 +451,10 @@ public void testInnerDelayedAuditOfLostBookies() throws Exception {
urLedgerList.contains(ledgerId));
Map<Long, String> urLedgerData = getUrLedgerData(urLedgerList);
String data = urLedgerData.get(ledgerId);
assertTrue("Bookie " + shutdownBookie
shutdownLatch.await();
assertTrue("Bookie " + shutdownBookieRef.get()
+ "is not listed in the ledger as missing replica :" + data,
data.contains(shutdownBookie));
data.contains(shutdownBookieRef.get()));
}

/**
Expand Down Expand Up @@ -503,7 +513,16 @@ public void testRescheduleOfDelayedAuditOfLostBookiesToStartImmediately() throws
urLedgerMgr.setLostBookieRecoveryDelay(50);

// shutdown a non auditor bookie; choosing non-auditor to avoid another election
String shutdownBookie = shutDownNonAuditorBookie();
AtomicReference<String> shutdownBookieRef = new AtomicReference<>();
CountDownLatch shutdownLatch = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie = shutDownNonAuditorBookie();
shutdownBookieRef.set(shutdownBookie);
shutdownLatch.countDown();
} catch (Exception ignore) {
}
}).start();

if (LOG.isDebugEnabled()) {
LOG.debug("Waiting for ledgers to be marked as under replicated");
Expand All @@ -522,9 +541,10 @@ public void testRescheduleOfDelayedAuditOfLostBookiesToStartImmediately() throws
urLedgerList.contains(ledgerId));
Map<Long, String> urLedgerData = getUrLedgerData(urLedgerList);
String data = urLedgerData.get(ledgerId);
assertTrue("Bookie " + shutdownBookie
shutdownLatch.await();
assertTrue("Bookie " + shutdownBookieRef.get()
+ "is not listed in the ledger as missing replica :" + data,
data.contains(shutdownBookie));
data.contains(shutdownBookieRef.get()));
}

@Test
Expand All @@ -547,7 +567,16 @@ public void testRescheduleOfDelayedAuditOfLostBookiesToStartLater() throws Excep
urLedgerMgr.setLostBookieRecoveryDelay(3);

// shutdown a non auditor bookie; choosing non-auditor to avoid another election
String shutdownBookie = shutDownNonAuditorBookie();
AtomicReference<String> shutdownBookieRef = new AtomicReference<>();
CountDownLatch shutdownLatch = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie = shutDownNonAuditorBookie();
shutdownBookieRef.set(shutdownBookie);
shutdownLatch.countDown();
} catch (Exception ignore) {
}
}).start();

if (LOG.isDebugEnabled()) {
LOG.debug("Waiting for ledgers to be marked as under replicated");
Expand All @@ -573,9 +602,10 @@ public void testRescheduleOfDelayedAuditOfLostBookiesToStartLater() throws Excep
urLedgerList.contains(ledgerId));
Map<Long, String> urLedgerData = getUrLedgerData(urLedgerList);
String data = urLedgerData.get(ledgerId);
assertTrue("Bookie " + shutdownBookie
shutdownLatch.await();
assertTrue("Bookie " + shutdownBookieRef.get()
+ "is not listed in the ledger as missing replica :" + data,
data.contains(shutdownBookie));
data.contains(shutdownBookieRef.get()));
}

@Test
Expand Down Expand Up @@ -664,7 +694,12 @@ public void testTriggerAuditorWithPendingAuditTask() throws Exception {
urLedgerMgr.setLostBookieRecoveryDelay(lostBookieRecoveryDelay);

// shutdown a non auditor bookie; choosing non-auditor to avoid another election
String shutdownBookie = shutDownNonAuditorBookie();
new Thread(() -> {
try {
shutDownNonAuditorBookie();
} catch (Exception ignore) {
}
}).start();

if (LOG.isDebugEnabled()) {
LOG.debug("Waiting for ledgers to be marked as under replicated");
Expand Down Expand Up @@ -715,7 +750,12 @@ public void testTriggerAuditorBySettingDelayToZeroWithPendingAuditTask() throws
urLedgerMgr.setLostBookieRecoveryDelay(lostBookieRecoveryDelay);

// shutdown a non auditor bookie; choosing non-auditor to avoid another election
String shutdownBookie = shutDownNonAuditorBookie();
new Thread(() -> {
try {
shutDownNonAuditorBookie();
} catch (Exception ignore) {
}
}).start();

if (LOG.isDebugEnabled()) {
LOG.debug("Waiting for ledgers to be marked as under replicated");
Expand Down Expand Up @@ -767,8 +807,17 @@ public void testDelayedAuditWithMultipleBookieFailures() throws Exception {
// wait for 10 seconds before starting the recovery work when a bookie fails
urLedgerMgr.setLostBookieRecoveryDelay(10);

// shutdown a non auditor bookie to avoid an election
String shutdownBookie1 = shutDownNonAuditorBookie();
// shutdown a non auditor bookie; choosing non-auditor to avoid another election
AtomicReference<String> shutdownBookieRef1 = new AtomicReference<>();
CountDownLatch shutdownLatch1 = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie1 = shutDownNonAuditorBookie();
shutdownBookieRef1.set(shutdownBookie1);
shutdownLatch1.countDown();
} catch (Exception ignore) {
}
}).start();

// wait for 3 seconds and there shouldn't be any under replicated ledgers
// because we have delayed the start of audit by 10 seconds
Expand All @@ -780,7 +829,16 @@ public void testDelayedAuditWithMultipleBookieFailures() throws Exception {
// the history about having delayed recovery remains. Hence we make sure
// we bring down a non auditor bookie. This should cause the audit to take
// place immediately and not wait for the remaining 7 seconds to elapse
String shutdownBookie2 = shutDownNonAuditorBookie();
AtomicReference<String> shutdownBookieRef2 = new AtomicReference<>();
CountDownLatch shutdownLatch2 = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie2 = shutDownNonAuditorBookie();
shutdownBookieRef2.set(shutdownBookie2);
shutdownLatch2.countDown();
} catch (Exception ignore) {
}
}).start();

// 2 second grace period for the ledgers to get reported as under replicated
Thread.sleep(2000);
Expand All @@ -793,9 +851,11 @@ public void testDelayedAuditWithMultipleBookieFailures() throws Exception {
urLedgerList.contains(ledgerId));
Map<Long, String> urLedgerData = getUrLedgerData(urLedgerList);
String data = urLedgerData.get(ledgerId);
assertTrue("Bookie " + shutdownBookie1 + shutdownBookie2
shutdownLatch1.await();
shutdownLatch2.await();
assertTrue("Bookie " + shutdownBookieRef1.get() + shutdownBookieRef2.get()
+ " are not listed in the ledger as missing replicas :" + data,
data.contains(shutdownBookie1) && data.contains(shutdownBookie2));
data.contains(shutdownBookieRef1.get()) && data.contains(shutdownBookieRef2.get()));
}

/**
Expand Down Expand Up @@ -825,7 +885,16 @@ public void testDelayedAuditWithRollingUpgrade() throws Exception {
// shutdown a non auditor bookie to avoid an election
int idx1 = getShutDownNonAuditorBookieIdx("");
ServerConfiguration conf1 = confByIndex(idx1);
String shutdownBookie1 = shutdownBookie(idx1);
AtomicReference<String> shutdownBookieRef1 = new AtomicReference<>();
CountDownLatch shutdownLatch1 = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie1 = shutdownBookie(idx1);
shutdownBookieRef1.set(shutdownBookie1);
shutdownLatch1.countDown();
} catch (Exception ignore) {
}
}).start();

// wait for 2 seconds and there shouldn't be any under replicated ledgers
// because we have delayed the start of audit by 5 seconds
Expand All @@ -838,7 +907,17 @@ public void testDelayedAuditWithRollingUpgrade() throws Exception {

// Now to simulate the rolling upgrade, bring down a bookie different from
// the one we brought down/up above.
String shutdownBookie2 = shutDownNonAuditorBookie(shutdownBookie1);
// shutdown a non auditor bookie; choosing non-auditor to avoid another election
AtomicReference<String> shutdownBookieRef2 = new AtomicReference<>();
CountDownLatch shutdownLatch2 = new CountDownLatch(1);
new Thread(() -> {
try {
String shutdownBookie2 = shutDownNonAuditorBookie();
shutdownBookieRef2.set(shutdownBookie2);
shutdownLatch2.countDown();
} catch (Exception ignore) {
}
}).start();

// since the first bookie that was brought down/up has come up, there is only
// one bookie down at this time. Hence the lost bookie check shouldn't start
Expand All @@ -856,11 +935,13 @@ public void testDelayedAuditWithRollingUpgrade() throws Exception {
urLedgerList.contains(ledgerId));
Map<Long, String> urLedgerData = getUrLedgerData(urLedgerList);
String data = urLedgerData.get(ledgerId);
assertTrue("Bookie " + shutdownBookie1 + "wrongly listed as missing the ledger: " + data,
!data.contains(shutdownBookie1));
assertTrue("Bookie " + shutdownBookie2
shutdownLatch1.await();
shutdownLatch2.await();
assertTrue("Bookie " + shutdownBookieRef1.get() + "wrongly listed as missing the ledger: " + data,
!data.contains(shutdownBookieRef1.get()));
assertTrue("Bookie " + shutdownBookieRef2.get()
+ " is not listed in the ledger as missing replicas :" + data,
data.contains(shutdownBookie2));
data.contains(shutdownBookieRef2.get()));
LOG.info("*****************Test Complete");
}

Expand Down
Loading