2 failed ESCs on bluejay 0.20.0

[jli]

Describe the issue

tl;dr I seem to have fried 2 new ESCs within ~1 hour of unremarkable flying, and the only thing that seems unique about my setups is that both were using Bluejay 0.20.0, so perhaps there's a bug...?

This may be total coincidence, but I built 2 new tiny whoops using the HappyModel ELRS F4 5A AIO flight controller, and both whoops seem to have bad ESCs within just ~10 packs/~1 hour of flying. I found this surprising because they both failed very quickly without any particularly nasty crashes. I acknowledge this may be total coincidence, or some other issue (maybe both FCs came from the same bad batch..?), but one more unusual thing about both of these builds is that I was using Bluejay 0.20.0, which I see was just released recently and seems to include some substantial changes/refactoring. So, I'm filing this just in case there is some bug that's causing hardware damage.

Some additional details:

• observed behavior: all motors except 1 spin up as expected in BF configurator. 1 stutters/twitches for a bit then stops. the behavior stays with ESC when i swap motors.
• hot motors: on one FC, during flying, i noticed the motor on the bad ESC got really hot, so I initially assumed the motor was bad. on the 2nd FC, i noticed the motor on the bad ESC got really hot while troubleshooting this problem via spinning up the motors in BF Configurator. oddly, this isn't reproducible, so i'm not really sure what's going on there.
• different versions: after the issue, i've tried flashing the ESCs to both bluejay 0.19 and the latest blheli-s (0.16.7 i think?), but the observed behavior remains the same, so i assume the hardware is damaged.
• other setup details: i'm using BF 4.4.2, dshot 300, bidirectional dshot, bluejay 0.20.0 48khz pwm, dynamic idle at around ~4-6krpm

Bluejay version

0.20.0

ESC variant

O_H_5

PWM frequency

48

DShot bitrate

300

Bidirectional DShot

On

FC firmware

Betaflight 4.4.2

Motor size

802, 702

Configurator debug log

No response

[damosvil]

Please, could you upload here a diff all and a photo of the builds?

[stylesuxx]

That's the same AIO that's being used in the Mobula6. I have flown dozens of packs with it during development without issues.

Hot motors are usually a sign of an imperfect tune. This does not explain why the motor got hot when bench testing, since PID loop is not interfering with the motor tab.

Can you visually see any damage on the FETs?

[jawz101]

Happymodel Mobula 7- after updating to v0.20.0 it eventually stopped arming where motors won't spin. I reverted to 0.19 and then back to blheli_s and now none of the firmwares won't spin motors. On the config page I did make sure the 1s vs 2s+ setting was set to 1s since the FC only supports 1s.

Coincidental that I came out here to look at the repo to see how the 0.20. build was working because I have a Mobula 7 as well and it's now no longer working at all.

[stylesuxx]

none of the firmwares won't spin motors.

So the motors spin or not?

• Can you see the ESCs still in esc-configurator or no?
• Can you spin the motors from Betaflight configurator?

Please be aware that the Mobula 7 originally comes flashed with BLHeli_M, so there actually is no "back to BLHeli_S".

PS.: In the future please open your own issue, otherwise it is really hard to keep track of the issues and users.

[jawz101]

I will post a separate issue

[stylesuxx]

Hey @jli - any update on this?

[jli]

Thanks @stylesuxx and @damosvil for the quick response, apologies for the delay.

Please, could you upload here a diff all and a photo of the builds?

Here are photos of the top and bottom of the 2 boards. The first board has a dead ESC for motor 1, and the second board has a dead ESC for motor 2. (Note that for my builds, I rotated the boards 90° CW and remapped the motors. I've marked the location of the dead ESCs in the photos to make it clear.)

photos of board with dead ESC 1:

photos of board with dead ESC 2:

Here are the configs.

• For dead ESC 1, I reflashed the board to try to get it work again, but I have an earlier config backup (made via the Presets tab "Save backup" button) that should be basically identical to when the ESC died: deadesc1_presetbackup__BTFL_cli_backup_mob6_memecam_20230720_004155.txt
• For dead ESC 2, I saved a diff all: deadesc2_diffall__BTFL_cli_mob7franky2_20230728_092956.txt

That's the same AIO that's being used in the Mobula6. I have flown dozens of packs with it during development without issues. [...] Can you visually see any damage on the FETs?

That's fair, I know it's a popular board. I'm not exactly sure what FET damage looks like, but I don't think I see any damage. All of the FETs on each of the boards look the same to me.

[stylesuxx]

Usually the broken FETs are easy to spot since they "pop", I circled the ones that look broken to me, they don't match your arrow though...

Profile 0 seems rather aggressive. I am personally not a fan of just using presets for tunes. Just because something says 1S or 65mm it does not fit everything. The diffs are very similar, so I am not overly surprised that the FETs went...

Software won't fix your issues at least with the first one where I am very confident to see popped FETs.

@damosvil asked for build pics in order to see the other components used - would you mind sharing which motors and props you used, just so that we can put the tune into a better context?

[jli]

Usually the broken FETs are easy to spot since they "pop", I circled the ones that look broken to me, they don't match your arrow though...

Yeah, those are the FETs by the plug for M3... I think that was some kind of residue on the FETs. I cleaned them off (just scraping with my fingernail), and it seems that got rid of it. Would a popped FET be able to be "cleaned up" that way?

Profile 0 seems rather aggressive. I am personally not a fan of just using presets for tunes. Just because something says 1S or 65mm it does not fit everything. The diffs are very similar, so I am not overly surprised that the FETs went...

Ah, yeah... I believe that profile is from the UAV-Tech 1-2S whoop preset. I don't recall if I was using that profile, or the more relaxed profile 1 or 2, but it's certainly possible I was using profile 0. Could you elaborate on what part of the tune is aggressive? The PID values, or other things like filters? (My usual tune is BF defaults + RPM filtering + dynamic idle, with D slider bumped up to 1.3. I think I'll go back to that..)

@damosvil asked for build pics in order to see the other components used - would you mind sharing which motors and props you used, just so that we can put the tune into a better context?

Yes sorry, I had scavenged the parts from those builds. I do have a photo of the build with dead ESC 1 though:

65mm build, motors are happymodel 702 26000kv, props are gemfan 31mm biblades. I think dry weight was ~20g.

I'm not as sure about the 2nd build, but I believe it was a 75mm build, with happymodel ex802 22000 or 25000kv motors, and gemfan 40mm biblades.

[stylesuxx]

Yeah, those are the FETs by the plug for M3... I think that was some kind of residue on the FETs. I cleaned them off (just scraping with my fingernail), and it seems that got rid of it. Would a popped FET be able to be "cleaned up" that way?

No, if it's popped, it's really mechanically damaged, you can't clean that - I'll see if I can find something in my parts bin as reference. Your clean version looks fine.

Your build seems sane, no crazy high KV motors, frame looks fresh, not over propped.

I assume the stuttering of the motor does not go away once you give more throttle, right? Im am just asking because 65mm 1S builds often like min/max startup power to be cranked up all the way or they might not start up smoothly on arm, but since it was once working, this is a long shot and probably won't change anything.

My usual tune is BF defaults + RPM filtering + dynamic idle, with D slider bumped up to 1.3. I think I'll go back to that...

Yes, I would highly recommend you to do that. If you change filtering or disable filters completely you should do short flights and check if the motors get overly warm, that is usually a good indicator that something is not right. In the best case you chose a FC with Blackbox so you have proper data to see what is happening.

This is a bit OT and heavily opinionated: I don't trust those AIO boards with built in VTX. They have all failed on me very, very quickly. Everything on the board is generating heat, adding a VTX which adds more heat is in my opinion really operating the hardware at it's maximum. I don't think that this was the culprit here, just something you might want to consider in the future. I like to go with an AIO that has a Blackbox instead and go with a separate VTX...

[jli]

I assume the stuttering of the motor does not go away once you give more throttle, right? Im am just asking because 65mm 1S builds often like min/max startup power to be cranked up all the way or they might not start up smoothly on arm, but since it was once working, this is a long shot and probably won't change anything.

No it doesn't go away. I would ramp up quickly from 0 to eg 1500 and the stuttering behavior was the same as ramping up slowly to values like 1100.

This is a bit OT and heavily opinionated: I don't trust those AIO boards with built in VTX. They have all failed on me very, very quickly. Everything on the board is generating heat, adding a VTX which adds more heat is in my opinion really operating the hardware at it's maximum. I don't think that this was the culprit here, just something you might want to consider in the future. I like to go with an AIO that has a Blackbox instead and go with a separate VTX...

Interesting, I've been doing more builds lately with separate VTX. Originally I was just motivated by having to pay less to replace broken parts, but it makes sense that more heat on a small board results in less reliability. Not to go even more OT, but I'm curious if you have particular components to recommend? I've used the happymodel cross f4 and the betafpv f4 (also cross shaped) for FCs and the happymodel ovx300 as the VTX. I'm currently evaluating other options for future builds, hoping to optimize for reliability 😅

[stylesuxx]

No it doesn't go away. I would ramp up quickly from 0 to eg 1500 and the stuttering behavior was the same as ramping up slowly to values like 1100.

Yeah, then the ESC is definitely gone. If you are adventurous you could try transplanting a set of good FETs from one board to the other.

I'm curious if you have particular components to recommend?

It's really difficult, I think everyone has their go to manufacturers. Personally I have had the best experiences with TuneRC - they have a cross FC with BB, but without RX and VTX, they might soon come out with the same one but including an ELRS RX. Other than that I have had good luck with the BetaFPV AIO cross that also comes with BB. Please take this with a grain of salt - I am not giving you a guarantee that it will work for you, but they have worked well for me and the Support is responsive.

[jli]

Yeah, then the ESC is definitely gone. If you are adventurous you could try transplanting a set of good FETs from one board to the other.

Yep, I'd like to learn how to do these types of repairs so I can give it a shot. Is it pretty much always the FETs? Is it surprising/odd that there's no visible damage to mine?

Thanks for all your help @stylesuxx 🙏

[schome1]

Although I can't confirm that v0.20 burned up three of my AIO FC ESCs, I can confirm that v0.20 was producing severe bounce-back during high-speed turns. I switched back to Bluejay v0.19 and the problem was gone. I've confirmed this on 8 TuneRC Poly F405 flight controllers. The two that used Bluejay v0.19 had no bounce-back issue. The others all had bounce-back issues during high speed turns while rolling left, and increasing throttle. All of the 8 builds had exactly the same hardware and electronics setup, using the same frames and the same props.

The three TuneRC Poly F405 flight controllers that burned up, all had Bluejay v0.20 installed, but I cannot confirm whether that was caused by the firmware or if it was because on those three flight controllers, I chose not to install a capacitor.

My drones were all configured with the following:

• Motors: 1202.5 11500KV
• Props: Geman 2512-3 and Emax Avan Rush 2.5"
• Batteries: 2S 550 mAh LiPo, 2S 660 mAh LiHV, 2S 720 mAh LiHV
• Weight without battery: 79.3 grams
• Battery weight range: 32 -35 grams
• VTX: HDZero Whoop Lite
• Camera: Runcam Nano 90
• RX: Happy Model ELRS EP2 TCXO

I hope this information helps.

[damosvil]

@schome1 please, could you provide blackbox data using 0.19 and 0.20? We also would need captures of esc-configurator.com settings, a diff all of the Betaflight configuration and a photo of the build to have more context about the conditions in which the issue happens.

[schome1]

@damosvil I will work on gathering that information this weekend.

[schome1]

requested-files.zip
The requested files have been uploaded. I was unable to reproduce the severe bounce-back that I had described, but while flying with Bluejay v0.20, I fried my fourth TuneRC Poly F405. The pattern is that all four have fried immediately after attempting a crash-flip. I don't hold the sticks down when I do a crash-flip. There were only 33 flights on it when it burned up. There were 17 flights on one of them, there were 10 on another, and 1 on another. I have other TuneRC Poly F405 flight controllers that have over 1000 flights combined, through many crash-flip modes, many races, many crashes. Maybe it's not v0.20, but that seems to be the common link between all of the burned flight controllers.

I hope you find the files useful. Please let me know if I can get any more data for you.

[damosvil]

@schome1 I have been checking the blackbox data, and I think that there is a reason for blowing the fets.

Version 0.19 updates extra telemetry data very often, every 16ms, so it is more prone to lose zero crossings and cut power to engage next zero crossing. For this reason 0.19 it is more forgiving on heating when handling agressive tunnings.

On the other hand version 0.20 updates extra telemetry data every 128ms, so it is much more robust on detecting zero crosings but if the tune is agressive it will try to do what it is asked to do, i.e. to update motor power as fast as possible. This causes motor heating, esc heating, and fet stress on agressive tunnings.

In addition to that what happens during crashes is that sometimes motor gets stuck, and esc desyncs. Unfortunately when this happens the fc tries to compensate increasing power on the motor to avoid tilting. If esc is not disarmed fast it heats and fets stresses.

Blheli_S and Bluejay up to 0.20 had a bug that caused power not to be limited during startup phases. So in the case the esc is not disarmed in a crash the fc may be sending high throttle to the esc, with a motor stuck and full power. So here we have the perfect storm, and the probability of killing escs is very high.

In a few days we will release 0.20.1, that correctly applies max startup power during motor startup. That will help on the crashes scenario. But you need to rebuild your tune.

Going back to blackbox data what I see is mainly fc is sending commands to the motors to correct gyro noise. This is normally caused by bad filtering and overagressive tunes.

I encourage you to:

• Apply Betaflight defaults and find the correct filtering settings to avoid injecting gyro noise in motors (this is what causes that motor handling looks like noise, instead motor handling curves)
• Rethink your tune to avoid motors correcting gyro noise.

This is the motor handling in your tunes:

This is the motor handling in a good tune:

[jli]

@damosvil Thanks for all those details!

Blheli_S and Bluejay up to 0.20 had a bug that caused power not to be limited during startup phases. So in the case the esc is not disarmed in a crash the fc may be sending high throttle to the esc, with a motor stuck and full power. So here we have the perfect storm, and the probability of killing escs is very high.

This is my understanding, could you confirm is this is correct?

• When arming for the first time, "max startup power" limits the amount of initial throttle command sent by the ESC to get the motor started. This helps protect the FETs from having too much power that can't be dissipated, in case there's something wrong (motor unable to spin for whatever reason).
• Whenever the ESC desyncs / the motor stops spinning, this is a "startup phase" since the ESC needs to start spinning the motor again.
• Blheli and Bluejay both do not apply the "max startup power" setting in this situation, so it may send a high throttle value and a lot of current through the FETs even if the motor is stuck. This affects Bluejay 0.20.0 also, but 0.20.1 will have a fix for this.

On the other hand version 0.20 updates extra telemetry data every 128ms, so it is much more robust on detecting zero crosings but if the tune is agressive it will try to do what it is asked to do, i.e. to update motor power as fast as possible. This causes motor heating, esc heating, and fet stress on agressive tunnings.

Just curious, what's the extra telemetry data? Is that like ESC temp?

[damosvil]

This is my understanding, could you confirm is this is correct?

* When arming for the first time, "max startup power" limits the amount of initial throttle command sent by the ESC to get the motor started. This helps protect the FETs from having too much power that can't be dissipated, in case there's something wrong (motor unable to spin for whatever reason).

When arming for the first time "max startup power" DO NOT limit the amount of initial throttle command sent to the ESC to get the motor started. ESC is NOT protected. There is a bug that causes this.
Fixed at: #156

* Whenever the ESC desyncs / the motor stops spinning, this is a "startup phase" since the ESC needs to start spinning the motor again.

Correct.

* Blheli and Bluejay both do _not_ apply the "max startup power" setting in this situation, so it may send a high throttle value and a lot of current through the FETs even if the motor is stuck. This affects Bluejay 0.20.0 also, but 0.20.1 will have a fix for this.

Correct. Not in this situation, neither when arming for the same time.

Just curious, what's the extra telemetry data? Is that like ESC temp?

https://github.com/bird-sanctuary/extended-dshot-telemetry

For now Bluejay sends relevant data in temperature frame, stress level frame and status frame.

[jli]

When arming for the first time "max startup power" DO NOT limit the amount of initial throttle command sent to the ESC to get the motor started. ESC is NOT protected. There is a bug that causes this.
Fixed at: #156

Hm, if max startup power wasn't limiting power on first arm or during desyncs, was it just never active?

[damosvil]

Hm, if max startup power wasn't limiting power on first arm or during desyncs, was it just never active?

https://github.com/bitdump/BLHeli/blob/20c36c2695c5b5ea855288e0046313688d1367a5/BLHeli_S%20SiLabs/BLHeli_S.asm#L1683

[damosvil]

@schome1 Please, could you try this filter settings and send me back a blackbox?

Also please, try reducing damping to 1.2 in pid tab.

[nuuls]

Hi! Does this also affect the power limit when crashing mid-flight? I've had it happen a few times where I bend a motor bell, and the motor is already smoked by the time I can disarm.

[stylesuxx]

Hi! Does this also affect the power limit when crashing mid-flight? I've had it happen a few times where I bend a motor bell, and the motor is already smoked by the time I can disarm.

Yes - also see here: #161

[ellmotion]

I also had an issue with a BetaFPV F4 1S 12A AIO FC V3 (Pavo Pico) after upgrading to 0.20.
I had a crash where the drone ended up upside down. I tried to use turtle mode to flip it back over and saw smoke in my goggles.
The FC was destroyed by the flames. I have no idea if this is a coincidence following the upgrade to 0.20.

Unfortunately, I am unable to retrieve any information from it because the USB interface has also been burned.

[stylesuxx]

The max startup fix in v0.20.1 should help in preventing such issues in the future. Turteling is a stress test for all involved components, if motors are blocked this can quickly lead to bad situations, especially when trying to push too far, too quick. Logs would have been great thou.

Sorry for your loss in any case...

[schome1]

@schome1 Please, could you try this filter settings and send me back a blackbox?

Also please, try reducing damping to 1.2 in pid tab.

Still with v0.20? I have been flying v0.19 with no issues this week. I have 108 flights in on this build. In my previous four builds (v0.20) I had a combined sum of 60 flights and all of them burned up.

Let me know if you want me to use v0.20 or if an update is available to try (I didn't see one when I checked). I'm going to wait to test with these settings until another flight controller arrives because I don't want to be left with none in the event that this one burns up too. It looks like it could arrive tomorrow, so it's very possible that I could test tomorrow or Sunday.

[damosvil]

Yes, with 0.20. The test would consist of trying those settings to check that if motor handling improved, but no need to crash, neither using turtle mode. Only 10 seconds of flight time to extract a blackbox and knowing if improving filtering and using less agressive pid settings are good for you.

[schome1]

bluejay-v0.20-pid-filter-test.bbl.zip
Here is the blackbox log you requested. I did change my frame - same size, same exact dimensions, just switched to a CF frame instead of G11. It's a little lighter. I did change my tune as well, and it's been flying great. I think the vibrations that were seen in the previous log files could have been from well used props at the time the log was saved.

This log is a short flight with Bluejay v0.20 and the PIDs/Filters as you suggested, plus new props.

[damosvil]

@schome1 many thanks for the log. We are double checking 0.20 code to improve startup protection. For now please stick to 0.19.2

[vad7]

CrazyF405HD after updating to 0.20 burned in flight:

[damosvil]

@vad7 If you are still be able to connect to the AIO, please, could you extract the blackbox data and upload it? Could you also let us know what motors, frame are you using and upload some photos of the build?

[hyp0dermik-code]

I see @schome1 was running dynamic idle
@ellmotion @vad7 did you have dynamic idle in BF enabled? A config dump or any log would tell us if you have one

[vad7]

yes.
Happymodel Mobula8 stock (EX1103 KV11000 9N12P)
BTFL_cli_backup_Mobula8_20230820_202604.txt

[ellmotion]

yes, dynamic idle was enable,
Unfortunately, the fc is out of service and I have no dump.
it was a pavo pico stock.
FC: F4 1S 12A AIO FC V3
Motors : BETAFPV 1102 14000kv

[jli]

just fyi for folks who've chimed in on this issue, it looks like 0.20.0 got pulled for potentially damaging ESCs/motors: https://github.com/bird-sanctuary/bluejay/releases/tag/v0.20.0

[k8ieone]

From what I'm reading here, that seems like a good move. I almost switched from BLHeli_m, but I happened to check open issues first. I'll wait until this issue is sorted out before I switch.

[damosvil]

https://youtu.be/RTcNqdShRx8?feature=shared
@vad7 I found the video of the flight. Was the kwad a custom build? Do you have any photo of the kwad before the incident?
I saw that when it took off core temp was 53 deg C. It suddenly raised over 60 deg C the last 3 seconds of the flight

It seems that the vtx was radiating heat over the mosfets, and resistor networks.
In that AIO the F405 MCU is in the opposite side of the vtx, and it was sensing 53 deg C. As I said mosfets and resistor networks are located in the side besides the vtx, so they were hotter for sure.

[vad7]

@damosvil Mobula8 Walksnail stock.
Walksnail Avatar HD Mini 1S Lite is very hot but cools down quickly during flight.
There were about 50 flights on firmware 0.19, the second flight on 0.20 ended like this.
Walksnail survived.

[stylesuxx]

I would like to encourage everyone in this thread to try our new RC of v0.20.1 - can be installed via esc-configurator.com and should fix all motor/esc protection issues

[hyp0dermik-code]

I would like to encourage everyone in this thread to try our new RC of v0.20.1 - can be installed via esc-configurator.com and should fix all motor/esc protection issues

When the startup power settings have been configured correctly by following https://github.com/bird-sanctuary/bluejay/wiki/Setup#startup-power-and-motor-idle-setup

[stylesuxx]

I would like to encourage everyone in this thread to try our new RC of v0.21.0 - can be installed via esc-configurator.com

Closing this issue down for the time being.