-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
deregistering process issues? #108
Comments
Hey @davidpan, I can't reproduce this 😕 Processes are correctly deregistered (deleted from |
Thanks for the reply. run bin/rails solid_queue:start
run bin/rails solid_queue:start --trace
environments: MySQL database version is: 8.0.35-0ubuntu0.22.04.1 (Ubuntu). Project-related environments: Project-related gems: solid_queue.yml file: default: &default
dispatchers:
- polling_interval: 1
batch_size: 500
concurrency_maintenance_interval: 300
workers:
- queues: "chrome_single"
threads: 1
processes: 1
polling_interval: 1
development:
<<: *default
test:
<<: *default
production:
<<: *default jobs : self.queue_adapter = :solid_queue
queue_as :chrome_single
limits_concurrency to: 1, key: ->(id) { id }, duration: 5.minutes The job is accessing web pages to collect content using chrome's headless mode. |
Thanks! That's very helpful as without the error I had no idea where to pull the thread from 😅 I've tried to reproduce this again, with a job that sleeps, but no luck. Could you try that again, but with a longer shutdown timeout, that you can configure as: config.solid_queue.shutdown_timeout = 60.seconds and right after issuing
And copy the output? |
After configuring for 60 seconds, retrying multiple times does not reproduce the error. run log:
SHOW ENGINE INNODB STATUS:
SELECT * FROM information_schema.innodb_trx:
|
Hey @davidpan, no, it shouldn't make a difference with respect to the processes registration and deregistration. That would only affect jobs in-flight, and in your case, you're still using up all the time:
My suspicion is that whatever transaction was open and locking your FWIW, I didn't touch |
Undo the previous configuration change: # config.solid_queue.shutdown_timeout = 60.seconds Retrying 2 times reproduces the problem. @rosa Running logs on the command line:
SHOW ENGINE INNODB STATUS:
SELECT * FROM information_schema.innodb_trx:
|
A test job that can be used to reproduce the problem. Press Ctrl-C when prompted. class CtrlCTestJob < ApplicationJob
self.queue_adapter = :solid_queue
queue_as :chrome_single
limits_concurrency to: 1, key: ->(i) { i }, duration: 5.minutes
def perform(i)
# Do something later
pp "start #{i}...."
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
web_driver = Selenium::WebDriver.for(:chrome, options:)
web_driver.manage.timeouts.implicit_wait = 20
web_driver.get "https://www.bing.com"
web_driver.manage.window.resize_to(1024, 1024)
pp web_driver.title
pp "press Ctrl-C "
sleep(10)
web_driver.close
web_driver.quit
end
end log:
|
Thank you! That's very helpful, I'll dig into these. |
@rosa I too am getting the "Error deregistering process" error. This has also resulted in unique constraint errors on claimed_executions when trying to start solid_queue. I still actually have the lingering processes in my database so I thought I'd do some poking around. When trying to destroy a process I get the following:
This in my local machine, a MacBook Pro. Using Postgresql 14 |
Oh, thank you so much @jmarchello! I think I know the problem, I'll try to work on a fix tomorrow, I didn't have time last week in the end. Thanks all for your patience 🙏 |
@rosa I may have realized that my particular issue was entirely my fault :). We use UUIDs for our ids so I had edited the initial solid_queue migration to use UUIDs, but I missed the |
Ahhhh! Thank you! 🙏 The infinite loop indeed didn't seem related to the error I think I have here (a race condition with the supervised processes and the supervisor trying to deregister) but it was interesting nonetheless! I was going to ask whether you perhaps had somehow a supervisor supervising itself 😅 |
@davidpan I finally could reproduce your issue, in my case I had to set config.solid_queue.shutdown_timeout = 2.seconds I'll work on a fix! |
Comment the following previously added. #config.solid_queue.shutdown_timeout = 60.seconds Modify gem source to github: gem 'solid_queue',:github => 'basecamp/solid_queue' execute bundle install
Running the test several times, pressing Ctrl-C before web_driver.get accesses the web address has a higher probability of success, and if you press Ctrl-C after accessing the web page, it has a higher probability of failure. A sample error log is shown below:
|
Thanks @davidpan! Could you try doing this?
Looks like you're still running the code without the fix in these logs above. |
@rosa Sorry, didn't realize it hadn't been merged yet. |
…ed (rails#108) * support queue pausing only if resque-pause is installed (for the resque adapter) * update readme * remove it from test descriptions
After running 'bundle exec rake solid_queue:start' for a while and stopping it with Ctrl-C, I noticed that the record of stopped processes in the 'solid_queue_processes' table was not deleted.
Upon restarting, I received an error message stating 'deregistering process 1 - {}' and the corresponding record still existed in the table.
The text was updated successfully, but these errors were encountered: