fix: No need to trigger ensembleChangeLoop if all failed bookies are not in current ensemble

[M1eyu2018]

Descriptions of the changes in this PR:

No need to trigger ensembleChangeLoop if all failed bookies are not in current ensemble.

Motivation

Fix the bug issue.
#4261

Changes

No need to trigger ensembleChangeLoop if all failed bookies are not in current ensemble.

Master Issue: #

---

In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
checks for pull requests. A pull request can only be merged when it passes precommit checks.

---

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

If this PR is a BookKeeper Proposal (BP):

• Make sure the PR title is formatted like:
  <BP-#>: Description of bookkeeper proposal
e.g. BP-1: 64 bits ledger is support
• Attach the master issue link in the description of this PR.
• Attach the google doc link if the BP is written in Google Doc.

Otherwise:

• Make sure the PR title is formatted like:
  <Issue #>: Description of pull request
e.g. Issue 123: Description ...
• Make sure tests pass via mvn clean apache-rat:check install spotbugs:check.
• Replace <Issue #> in the title with the actual Issue number.

---

[hangc0276]

To fix this issue, check the delayedWriteFailedBookies before replace can solve this problem. But it is better to failed the write request when the write request received failure callback.

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java

Lines 1210 to 1212 in e6b4154

	} else {
	nettyOpLogger.registerFailedEvent(MathUtils.elapsedNanos(startTime), TimeUnit.NANOSECONDS);
	}

@horizonzy tested this solution, and it works.

[horizonzy]

#4285 can fix the problem, but this PR is also good for avoiding the unnecessary ensembleChangeLoop. I think this is useful.

[M1eyu2018]

@hangc0276 @horizonzy I have added a unit for this PR. PTAL

[lhotari]

I this related to #4194? /cc @shustsud

[shoothzj]

LGTM

[shoothzj]

@dlg99 @hangc0276 @eolivelli PTAL, thanks

[hangc0276]

We need to set changingEnsemble = true when entering the else code block to avoid a potential race condition. If all the bookies in the toReplace set do not exist in the current ensemble, set changingEnsemble back to false.

What's more, please add a comment to explain why we need this logic. thanks.

[hangc0276]

@M1eyu2018 Thanks for your contribution. Would you please help check the failed tests? Thanks.

[shoothzj]

Sorry I didn't remember why I close this last week. It maybe a mis-operation. @M1eyu2018 Would you please help check the failed tests? Thanks.

[M1eyu2018]

Sorry I didn't remember why I close this last week. It maybe a mis-operation. @M1eyu2018 Would you please help check the failed tests? Thanks.

OK, I will check as soon as possible.

[M1eyu2018]

@hangc0276 @shoothzj
I have checked the failed tests.