Holybro ESC problems (can't fly with master) #13147

dakejahl · 2019-10-09T18:43:42Z

Describe the bug
Cannot fly my holybro s500 kit with master. It works fine on v1.9.2. I've also tracked this on px4 discuss here.

To Reproduce
Using this holybro s500 kit: https://shop.holybro.com/_p1153.html

Expected behavior
There should be no ESC issues between PX4 firmware versions. There is funny business happening on the actuator outputs during boot.

Below are screenshots of a scope on motor output 1 during first 1 second of boot.

Log Files and Screenshots
v1.9.2

master

As you can see in the images above, v1.9.2 has 5 pulses of 900us width spaced at 20ms before the 400hz 900us outputs actually kick on. I am not sure if this is related, but it was the only thing I found.

Additional context
This has been broken for a while now, I'd really like to be able to fly master...

The text was updated successfully, but these errors were encountered:

julianoes · 2019-10-10T09:44:16Z

@bkueng have you seen this?

davids5 · 2019-10-10T20:27:33Z

I have requested hardware so that I can help with this. but I will not be able to work on it till after I return on the 16th.

MaEtUgR · 2019-10-14T10:51:43Z

We should have an assembled set. Let me check.

EDIT: It's in use, that would mean ours is still flying on master. I'll have to check when it's available again.

dagar · 2019-10-14T16:44:38Z

ould mean ours is s

@MaEtUgR could you try through a range of typical configurations?

400 Hz PWM
OneShot
DShot

dakejahl · 2019-10-14T22:48:47Z

Considering no one else but me is having this issue, I wonder about the ESC firmware. This might be something we ask Holybro about. Also Holybro has multiple ESC models, and I am not sure which model these are, the website does not list it.

jkflying · 2019-10-23T15:59:47Z

It would be helpful to get a high-rate log, set this by enabling the flag in SDLOG_PROFILE, get a log with 1.9.2, do the same on master, and then upload the logs to compare.

MaEtUgR · 2019-10-23T17:19:05Z

I'm currently trying to reproduce. Attached one of these black heat shrink ESCs of an S500 to a pixhawk 4, chose quad x and tried force arming and running pwm test. First result: The signal is on IO by default but @dakejahl mentioned he has the problem on fmu so I should probably load the S500 airframe config.

dakejahl · 2019-10-23T17:23:57Z

@dakejahl mentioned he has the problem on fmu

Sorry I think I misunderstood the question on the dev-call, I am indeed connected via IO PWM OUT

MaEtUgR · 2019-10-23T18:04:05Z

Finally I got it working, I had some old custom mixer on the SD card of the flight control that screwed me up. Now I tried spinning the motor without prop on 1.9.2 and master and it looks exactly the same to me. I can do further tests tomorrow.

MaEtUgR · 2019-10-23T18:17:07Z

Here's a picture of the setup just for sanity check by @dakejahl :

dakejahl · 2019-10-23T18:42:33Z

Yup looks like mine. I opened up the heat shrink and unfortunately there's not really much silk screen info.

dakejahl · 2019-10-23T18:43:40Z

Is there anyone from HolyBro that can join this conversation? I'm like 90% sure this is a firmware bug that they probably know about and have fixed if we have the same hardware and yours works and mine doesn't.

dakejahl · 2019-10-23T23:26:18Z

Using the pwm test -p 1600 -a works great, motors all spin correctly. I'll post a log of highr rate data once I am able to test with master. I am currently blocked by an erroneous "heading estimate not stable, preflight checks failed" on master ... not sure why because if I flash 1.9.2 I can fly around with no issues.

MaEtUgR · 2019-10-24T15:09:13Z

"heading estimate not stable, preflight checks failed"

had it once as well, @bresch fixed it in #13261

If it's changing from 1.9.2 to master it has to be a software caused problem. Currently I think the vibrations getting amplified is the most plausible one since I didn't have the fc mounted to the fram -> no vibrations and your pwm test also runs fine. You could try throttling up a bit with much lower MC_(ROLL,PITCH,YAW)RATE_K gain and check if your problem is gone. Be aware that you might not be able to fly in a stable fashion with much lower gains.

@dakejahl It would be really helpful to debug if you could like @jkflying suggested provide a high rate log and a short video showing how the symptom actually looks like such that we are sure it's there are no misunderstandings.

dakejahl · 2019-10-24T17:08:08Z

@MaEtUgR I've got the FC isolated on a little vibe isolation mount with damping bulbs but I think you're correct.

1.9.2
You can actually still see the effects of vibration on the motor output at high throttle. And I can indeed hear the bad sounds the motors make near high throttle.

https://review.px4.io/plot_app?log=7f7aba71-125c-4861-9b67-6ced2962f7a1

master
Yup this is totally messed up. I am surprised I couldn't see this with low rate logging.

https://review.px4.io/plot_app?log=dccabbd4-65cd-4efd-bd10-421a9504dfb4

I guess my big question is: Why? Is this a consequence of running the rates controller at a higher frequency? What can I do to fix this? Why am I the only person who has experienced this? (does everyone else have baller vibration damping? 😉 )

julianoes · 2019-10-25T11:51:16Z

@dakejahl that looks like crazy vibrations on gyro and accel, wow! I don't understand where it comes from. Is there a chance you could do a git bisect to try to figure out when it was introduced?

dakejahl · 2019-10-25T16:47:26Z

Yes I will investigate this early next week when I am back in town.

dakejahl · 2019-10-30T23:56:23Z

Ha... of course the very first commit from git bisect doesn't even work. Can't arm, vehicle doesn't see RC.

ea48cd4 looks like px4io is failing to start on this commit. Enabled it manually and am able to arm, but the motors don't spin so something else is wrong here as well.

Any tips?

julianoes · 2019-10-31T15:21:35Z

@dakejahl you can do git bisect skip and try the next commit.

dakejahl · 2019-10-31T17:35:13Z

Sad day... while testing during a bisect, the vehicle flipped itself and smashed on the ground destroying two props. Unfortunately I need to order some more, my spares are incompatible with these motors.
https://review.px4.io/plot_app?log=a7f25022-d98f-4cc4-b89b-12cda7d77760

dakejahl · 2019-10-31T21:06:29Z

Here's a bench test with no props attached, I arm the vehicle and let it idle for a little bit, then ramp up the throttle to 0.5 and then back down to idle.

1.9.2
https://review.px4.io/plot_app?log=bf9f4248-a7e5-42dd-9f83-e60df970ad0a

master
https://review.px4.io/plot_app?log=eccaa7ec-6fb8-4e93-88dd-9689243c1c05

You can clearly see the raw gyro/accel go absolutely nuts on master as soon as there is more than idle throttle.

Now the hard part here is determining cause/effect. I did a diff git diff v1.9.2 master -- src/drivers/imu/mpu6000/ and nothing changes functionally with the driver (20689 is crammed in with mpu600). I verified on the bench that the sensor is being selected correctly and that the data more or less looks the same when you poke it. Also notice the vibration levels between 1.9.2 and master are pretty much the same at idle throttle, the data really only gets bad when you throttle up.

This leads me to believe the crazy vibrations are real vibrations -- caused by the shuddering of the motors (Video of shuddering). If you take a closer look at the actuator outputs, you can see on master the output values do not increase smoothly, but are rather jagged.

1.9.2

master

dakejahl · 2019-10-31T22:34:09Z

Changing IMU_GYRO_CUTOFF from 80 back to 30 solved the issue 😕

I can't remember why I changed it away from default in the first place.... 🤷‍♂️ (I think it was early on during bringup and it was flying like crap and someone suggested to change the gyro cutoff). But still concerning that there is such a large difference in behavior between two versions. Any ideas?

https://review.px4.io/plot_app?log=b214a366-11a8-4ca3-9850-0faa3fd62c57

LorenzMeier · 2019-11-01T07:17:16Z

That's not really concerning because we've changed the filtering pipeline to accommodate high-performance racers so that the cutoff setting is more effective. I'll let @bkueng / @bresch chime in, but that frame has a lot of flex and so what I'm seeing here is what I would expect with 30 vs 80 Hz cutoff.

mrpollo · 2019-11-03T15:09:45Z

Thanks for doing the extra work to verify this @dakejahl.

@bkueng @bresch, please close this issue if you can verify @LorenzMeier conclusion, we need to move on with the v1.10 release.

bresch · 2019-11-03T20:48:40Z

@mrpollo I think we still need to find the cause of the issue, lowering the cutoff just hides the underlying possible issue.
This is why the bisect is really important to helping us finding the modifications that leads to the change of behavior.

@dakejahl any news?

dakejahl · 2019-11-03T21:26:42Z

Unfortunately I can't continue the bisect with the s500 until I get some new props. I'm going to setup my 250 airframe in the mean time and see if I can't reproduce the issue. Travelling this week so I'll be back on it the following week.

…

On Sun, Nov 3, 2019, 1:49 PM Mathieu Bresciani ***@***.***> wrote: @mrpollo <https://github.com/mrpollo> I think we still need to find the cause of the issue, lowering the cutoff just hides the underlying possible issue. This is why the bisect is really important to helping us finding the modifications that leads to the change of behavior. @dakejahl <https://github.com/dakejahl> any news? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13147>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI27PPRADRY6GPYL2EVJWUDQR42MDANCNFSM4I7DKCIQ> .

julianoes · 2019-11-05T13:48:25Z

Here is another example of a frame with a lot of vibration and presumably overdriven motors:
https://discuss.px4.io/t/quad-crash-at-manual-mode/13579/3

mhkabir assigned davids5 Oct 9, 2019

julianoes added Drivers 🔧 Sensors, Actuators, etc bug labels Oct 10, 2019

weekly-digest bot mentioned this issue Oct 13, 2019

Weekly Digest (6 October, 2019 - 13 October, 2019) #13171

Closed

dagar added this to the Release v1.10.0 milestone Oct 14, 2019

julianoes removed this from the Release v1.10.0 milestone Nov 6, 2019

julianoes unassigned davids5 Nov 6, 2019

dakejahl closed this as completed Jan 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Holybro ESC problems (can't fly with master) #13147

Holybro ESC problems (can't fly with master) #13147

dakejahl commented Oct 9, 2019

julianoes commented Oct 10, 2019

davids5 commented Oct 10, 2019

MaEtUgR commented Oct 14, 2019 •

edited

Loading

dagar commented Oct 14, 2019

dakejahl commented Oct 14, 2019

jkflying commented Oct 23, 2019

MaEtUgR commented Oct 23, 2019 •

edited

Loading

dakejahl commented Oct 23, 2019 •

edited

Loading

MaEtUgR commented Oct 23, 2019

MaEtUgR commented Oct 23, 2019

dakejahl commented Oct 23, 2019

dakejahl commented Oct 23, 2019

dakejahl commented Oct 23, 2019 •

edited

Loading

MaEtUgR commented Oct 24, 2019 •

edited

Loading

dakejahl commented Oct 24, 2019 •

edited

Loading

julianoes commented Oct 25, 2019

dakejahl commented Oct 25, 2019

dakejahl commented Oct 30, 2019 •

edited

Loading

julianoes commented Oct 31, 2019

dakejahl commented Oct 31, 2019

dakejahl commented Oct 31, 2019 •

edited

Loading

dakejahl commented Oct 31, 2019 •

edited

Loading

LorenzMeier commented Nov 1, 2019

mrpollo commented Nov 3, 2019

bresch commented Nov 3, 2019

dakejahl commented Nov 3, 2019 via email

julianoes commented Nov 5, 2019

Holybro ESC problems (can't fly with master) #13147

Holybro ESC problems (can't fly with master) #13147

Comments

dakejahl commented Oct 9, 2019

julianoes commented Oct 10, 2019

davids5 commented Oct 10, 2019

MaEtUgR commented Oct 14, 2019 • edited Loading

dagar commented Oct 14, 2019

dakejahl commented Oct 14, 2019

jkflying commented Oct 23, 2019

MaEtUgR commented Oct 23, 2019 • edited Loading

dakejahl commented Oct 23, 2019 • edited Loading

MaEtUgR commented Oct 23, 2019

MaEtUgR commented Oct 23, 2019

dakejahl commented Oct 23, 2019

dakejahl commented Oct 23, 2019

dakejahl commented Oct 23, 2019 • edited Loading

MaEtUgR commented Oct 24, 2019 • edited Loading

dakejahl commented Oct 24, 2019 • edited Loading

julianoes commented Oct 25, 2019

dakejahl commented Oct 25, 2019

dakejahl commented Oct 30, 2019 • edited Loading

julianoes commented Oct 31, 2019

dakejahl commented Oct 31, 2019

dakejahl commented Oct 31, 2019 • edited Loading

dakejahl commented Oct 31, 2019 • edited Loading

LorenzMeier commented Nov 1, 2019

mrpollo commented Nov 3, 2019

bresch commented Nov 3, 2019

dakejahl commented Nov 3, 2019 via email

julianoes commented Nov 5, 2019

MaEtUgR commented Oct 14, 2019 •

edited

Loading

MaEtUgR commented Oct 23, 2019 •

edited

Loading

dakejahl commented Oct 23, 2019 •

edited

Loading

dakejahl commented Oct 23, 2019 •

edited

Loading

MaEtUgR commented Oct 24, 2019 •

edited

Loading

dakejahl commented Oct 24, 2019 •

edited

Loading

dakejahl commented Oct 30, 2019 •

edited

Loading

dakejahl commented Oct 31, 2019 •

edited

Loading

dakejahl commented Oct 31, 2019 •

edited

Loading