The bot is running out of CPU credits

[HebaruSan]

Problem

The status page claims that all sorts of things have been indexed over the past two days:

But that doesn't square with the commits in CKAN-meta:

On the forum the specific examples of PAWS and Trajectories were given:
https://forum.kerbalspaceprogram.com/index.php?/topic/154922-ckan-the-comprehensive-kerbal-archive-network-v1262-dragon%EF%BB%BF/&do=findComment&comment=3621485

And indeed, they both have new releases on their repos, the bot says it indexed them when they came out, but there's no new .ckan file.

So the bot is running, and it's finding new releases and updating its status file, but they're not getting indexed. Is there something weird with its git config that's stopping changes from being pushed to GitHub?

[techman83]

We're reaching another capacity limit for our brute force method of indexing.

The box is out of CPU credits. So it will be taking longer than the allowed timeout value and terminating the run before it has finished.

I have been pondering lately about how to better re-architect this, because whilst it has been a pretty robust service, it's expensive spinning up the Mono runtime ~1500 times and doing all the processing around it. Ideally it would be really nice to write some python bindings like you would with C/C++, but I'm unsure how doable that is with C# and if how much would even be possible. At the least, translation/validation/final output would ideally be done by netkan, even if Python did all the web calls etc.

Also the deployment is very manual, so making small changes and deploying them is time consuming, which makes it hard to iterate quickly.

TLDR

I've disabled the indexer and I'll turn it back on tonight. Along with making the runs every 6 hours now.

[HebaruSan]

Huh, I always assumed it would be RAM or disk space that sank us.

Would it help if we had a netkan --bot command, which performed one whole bot pass in one instance of Mono?

[DasSkelett]

That lite_index option would be handy too probably.

[techman83]

I got carried away as I've kinda wanted to re-architect it for some time.. KSP-CKAN/CKAN#2789

[techman83]

So adjusting things to 8 hourly allows the credits just enough time to recover before the next run. And the box has been running with no spare credits for several weeks, so it must have only just recently been taking longer than 3 hours to finish a run (which I've just monkey patched to 6 hours).

[HebaruSan]

The status quo would be fine with one simple change: Suspend/resume when credits run out instead of terminating. Is there by any chance a server option for that?

[techman83]

@HebaruSan - That's not an entirely silly idea. I could write a little awscli script that checks if there are more than X credits available, and the last run was outside the total run time. I'll check what IAM permissions the instance needs to pull that information.

I also had a think about the queuing. Essentially it would be a case of adding the SDK (that might be hard or easy), setting the auth environment variables and with long polling open a receive message request on a loop and do something with the results.

https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/EnableLongPolling.html

[HebaruSan]

This issue was about the bot not working, and it is functioning now. There are still things we would like to improve, but that's what KSP-CKAN/CKAN#2789 is for.

[DasSkelett]

Well actually, now it's the other way around, the commits are pushed, but it appears that the status page isn't updated?
The status page claims the last indexed mod is TarsierSpaceTechnologyWithGalaxies, but since then there have been more commits:
https://github.com/KSP-CKAN/CKAN-meta/commits/master.

Sorting for last inflated / last checked tells me that it stopped updating the page at the end of the run (ZZZRadioTelescope).
Has this something to do with KSP-CKAN/NetKAN-status#8 @techman83?

[HebaruSan]

Those commits came from the SpaceDock web hook rather than the bot.
The part of KSP-CKAN/NetKAN-bot#80 where the web hook was supposed to update the status page does not work, I think because the web hook does not have access to the same status file.

[techman83]

Hmm, I thought that did work? We'll fix it properly with the re-architecture.

[HebaruSan]

it is functioning now

I spoke too soon, the bot hasn't gotten to PoodsCalmNebulaSkybox in 2 days.

[techman83]

/me sighs, yup. I've disabled it and killed the current run. I'll whip up a credit checker over one of my coffee breaks today.

[techman83]

Think I just found an issue with NetKAN, I'll raise an issue.

[HebaruSan]

Working again thanks to #7269.
KSP-CKAN/CKAN#2816 should fix the underlying issue (this time).