RMD is Switching Languages (TypeScript)

[shadowmoose]

Update on RMD and Development Progress

Hey everybody interested in RMD's development progress. I'm throwing this discussion thread together in order to give an update on RMD.

You may have noticed that over the last few months (or years), that development on RMD has slowed or stalled. While this is largely due to RMD being (almost) feature-complete, there have been more technical issues at work.

The fact of the matter is: RMD is built in Python, and python kinda sucks for this use case. Originally, when RMD was a simple custom single-file script for personal use, speed and performance were irrelevant. Now, however, RMD has blossomed into a multi-process beast, splitting its download and processing workload into multiple OS processes in order to use multiple CPU cores - and also to not hang while downloading data. All of these processes eat a ton of resources while running, have problems concurrently accessing the shared database (manifest), and require quite a lot of wiring and boilerplate just to let them communicate efficiently.

Additionally, RMD's UI is web-based - which means it requires JavaScript. This has the effect of making it difficult to contribute anything meaningful to the project as an outsider, because your change may require you knowing both languages! While there are other methods one could use to approach solving some of these issues in Python, there are languages built to work with this exact scenario much more fluidly.

Enter, TypeScript

TypeScript, running via Node, is specifically built for this exact type of use case. Additionally, since RMD's UI already uses React (JS), a conversion to TS allows RMD to use the same language all the way down the stack. Aside from having a mono-language project again, there are a few more major advantages:

• JS's async nature fits naturally with RMD's workload of primarily IO operations.
• Stronger typing makes it orders of magnitude simpler to pick up and work on casually.
• More battle-tested libraries which overlap RMD's specific needs.
• Simpler bundling/releasing - it can be trivial to pack Node code, using any number of tools.
• The ability to use event-based code, for a much more responsive download progress update.

While I'm already in the process of rebuilding the app from scratch, it only makes sense to finally add some major changes to RMD's operation. Things like source groups, better file management, and an an extensible database design. This should allow the (currently extremely rigid) UI to be more flexible. I expect that this migration will make it much simpler to add new features such as text-only downloading/browsing, searching for specific file metadata, and real-time input from the interface.

The Bad News

The downside of this change in RMD is that it simply will not be compatible with previous version's manifests. I wish that it were possible - as with past releases - to support some form of automatic migration... However, in this case, the programs are simply going to be too different. The original python-based RMD only stores and tracks about half of the post data that typescript-RMD will be able to use.

That said, there's no reason you can't continue to use python-RMD either. While it won't be supported once RMD-TS is completed, the existing code should function fine as-is, like it has already been doing for years. I also have no plans of wiping the original Python code out, so there will be a py branch that other contributors are welcome to add to if they wish.

Interested in More?

Either open a thread in these Discussions, or check out the WIP code over in the rmd-typescript branch. If you have any feedback or ideas, please feel free to request them here as well.

[commanderflatus]

So how's it going? I'm eager to try out the new version.... I'm a longtime user of this great software, excited for new developments....

[shadowmoose]

Sorry about the long absence. I've been extremely busy lately, and this project had taken the back-burner.

I'm expecting to have some free time to look at this again soon. It's almost done, and is already in a runnable state for the Sources that are currently built in. At this point the branch mostly needs the remaining sources added, and then some extensive testing with real-world scenarios. After that, I have to figure out how (or if) the new version will be packaged together. This last step could take a while, or could be trivial, depending on how support for bundling library binaries has progressed since I last ran into those issues.

Once that's all done, there are a few new features I was toying around with locally that I'd like to add, but I'm not going to commit to anything new until the basic features are complete. All this is to say that the rewrite is working excellently, and I hope to have time to wrap it up soon. If anybody who knows TypeScript is eagerly anticipating the release, PRs are always welcome to speed things along.

[Kristijan1001]

Nice, Is the new version going to support Pushshift and stuff for old reddit posts still ?

[shadowmoose]

Yes, the rebuild should support all of the original features and sources - provided that they all still exist and function.

[varadins]

Looking forward to this!

[ghost]

Any updates ?