Problem with the filtered RSS feeds

[daveverwer]

The Recent Releases RSS feed is working, but I've noticed problems with the filtered RSS feeds. I subscribe to four different feeds:

• https://swiftpackageindex.com/releases.rss?major=true
• https://swiftpackageindex.com/releases.rss?minor=true
• https://swiftpackageindex.com/releases.rss?patch=true
• https://swiftpackageindex.com/releases.rss?pre=true

For the last few weeks, I've noticed only a small number of releases in each of these feeds as I do my regular Thursday trawl through my RSS. For example, today looked like this:

I've just looked into it, and I think it may be because of the way we list packages in the recent_releases view. It only has 100 items total across all types of package releases, and then filters are narrowing that down. I've just verified with the raw feed data, and the major version feed only contains 7 total items as only 7 of the last 100 packages were major versions.

A potential fix would be for the view to return significantly more items than are needed, then apply the limit of 100 after the releases have been filtered.

What I don't quite understand is why my feed reader isn't coping with this. It's supposed to refresh every hour and cache items that roll off the end of the feed. The feed itself is certainly exhibiting this problem, but my reader should cope with that situation. I'm creating a discussion rather than an issue as I need to figure out why that's happening.

We should also increase the number of items in the RSS feed. At the current time, the last 100 releases only covers five days. We should have it at 200 or 300 even.

[daveverwer]

Yea, there's definitely something funny going on. Since yesterday, my feeds only show one new release:

Whereas these are the tweets since yesterday:

• https://twitter.com/SwiftPkgUpdates/status/1458915196089602049
• https://twitter.com/SwiftPkgUpdates/status/1458924052400287746
• https://twitter.com/SwiftPkgUpdates/status/1458926477702049793
• https://twitter.com/SwiftPkgUpdates/status/1459022466374905888
• https://twitter.com/SwiftPkgUpdates/status/1459060829177921536
• https://twitter.com/SwiftPkgUpdates/status/1459064800059834382
• https://twitter.com/SwiftPkgUpdates/status/1459124797573677056

[daveverwer]

OK. I'm starting to debug this properly but it'll take a few days to figure out what's going on.

I am subscribed to the following feeds:

• https://swiftpackageindex.com/packages.rss
• https://swiftpackageindex.com/releases.rss
• https://swiftpackageindex.com/releases.rss?major=true
• https://swiftpackageindex.com/releases.rss?minor=true
• https://swiftpackageindex.com/releases.rss?patch=true
• https://swiftpackageindex.com/releases.rss?pre=true

I'm subscribed in NetNewsWire locally, and through Reeder via InoReader server-side polling.

As of 3:30 pm today, everything is marked as read in all RSS feeds across both platforms.

Already there's something weird, though. The latest releases on the website show like this:

The most recent release is 4 hours ago, yet there have been three tweets since that release:

I'm going to report back on the situation tomorrow on how all feeds have updated in 24h.

[daveverwer]

So, 24 hours in and there are several issues being exposed!

The first issue logged as #1410.

The second issue is that we're using the createdAt date for the package for the pubDate which may be wrong and we might be better using the releasedAt date, but we only want to make one change at a time! I won't write this up into an issue until this first step has been tested.

[daveverwer]

I've been monitoring these new feeds since #1410 was deployed and everything looks really good, with everything behaving correctly.

However, 100 items now covers less than 5 days of releases at the current rate, and that's only going to get worse as we go forward so I think we should make one more change in #1443.