generate_emu_config overhaul plus minor tweaks and fixes

[alex47exe]

• [alex47exe] major overhaul of generate_emu_config - custom configs, proper ini parsing, better logging and error handling, helper tools:

  • add -def1 ... -def5 arguments, which can be used to generate your preferred custom config
    if no -def argument is provided, -def1 will be used by default, to automatically copy from the following folders:

    • .\_DEFAULT\0 ............... essential emu files, like latest GSE dlls (steam_api.dll and steam_api64.dll)
    • .\_DEFAULT\1 ............... other GSE files and folders, including default ini files
    • .\_DEFAULT\<appid> ... other GSE files and folders, but only for the current <appid>, if the folder exists

  • (Windows only) add some useful helper tools, written in AutoIt3:

    • gse_acw_helper.exe - add the required achievements schema db files for Achievement Watcher, if .\steam_misc\extra_acw\extra_acw.zip file exists (if generated by generate_emu_config.exe -acw <appid>)
    • gse_debug_switch.exe - automatically switch between release and debug versions of the emulator, if steam_api.7z / steam_api64.7z file exists (or steamclient.7z / steamclient64.7z, if you use the steamclient version)
      paths to release and debug files inside 7z, can be customized in .\steam_misc\tools\au3\scripts\gse_debug_switch.ini
    • gse_generate_interfaces.exe - simple x64-x86 launcher for generate_interfaces.exe
      it also writes all found steam interfaces to CODEX steam_emu.ini (if generated by generate_emu_config -cdx <appid>)
      make sure to name your original dll to one of these formats, so it can automatically find its interfaces:
      • valve_api.dll / valve_api64.dll
      • steam_api.dll.bak / steam_api64.dll.bak or steam_api.dll.org / steam_api64.dll.org
      • steam_api.bak / steam_api64.bak or steam_api.org / steam_api64.org
      • steam_api_orig.dll / steam_api64_orig.dll or steam_api_legit.dll / steam_api64_legit.dll
    • gse_lobby_connect.exe - simple x64-x86 launcher for lobby_connect.exe

  • new folder structure, compatible with current and future helper tools --- default arguments are -acw -cdx -clr <appid>
    NEVER delete .\steam_misc\app_backup, .\steam_misc\app_info, .\steam_misc\tools and .\steam_settings folders
    MIGHT need .\steam_misc\extra_acw and .\steam_misc\extra_cdx for compatibility with Achievement Watcher and CODEX

  • add -scx argument to automatically download images / videos for trading cards, backgrounds, badges, emoticons and other tradable items
    unfortunately I couldn't find any direct steam api method to download the files, so I had to write a rudimentary web scrapper to extract the download links from a third-party website, hence the scx_gen.py script might need updating in the future if the website design changes

  • download screenshots and videos:

    • download thumbnails for both screenshots and videos, and compress them to .zip files
    • screenshots and videos are now numbered from first to last published, as in the Steam store page
    • add -vids_low / -vids_max arguments to download all videos, in low and / or high quality

  • create / update .\top_owners_ids.txt when .\top_owners_ids.html is present

  • generate controller action sets txt files for all found controller vdf configs, and zip them inside .\steam_misc\app_backup\app_backup.zip
    by default, the emu supports only xboxone and xbox360 controller configs, though if the're are any issues with the default supported controller action sets inside .\steam_settings\controller folder, you could try to unpack and overwrite action sets for other unsupported controller configs

  • (Windows only) add AdvancedRun launchers (cmd console + silent) for .bat files and .py scripts

• [alex47exe] major overhaul of migrate_gse - uses the same .\_DEFAULT\0 and .\_DEFAULT\1 folder structure for default configs
  it can convert old .txt format to .ini format, minus branches.json, which would require using top_owners_ids.txt and some login code from generate_emu_config, which should actually be used to properly generate the config files, instead of converting from the old .txt format

• [alex47exe] generate_interfaces.exe - find all Steam Interfaces instead of only old ones
  the emu will ignore the ones it doesn't require, while we'll have the complete list to write it to CODEX steam_emu.ini

• [alex47exe] lobby_connect.exe - improve cmd console text alignment

• [alex47exe] mods_img instead of mod_images (better folder consistency), better example for mods_img, minor tweaks to .ini and .md files

[alex47exe]

Not sure why it takes so long to build the experimental debug api here. Mine was done in less than 3 minutes:
https://github.com/alex47exe/gse_fork/actions/runs/10154145974/job/28078796689
Maybe you could try to re-run checks again. Or try a merge to a different branch, and run 'Prepare release' action manually.

[Detanup01]

Seems like github issue (I could merge without actually waiting to finish all builds btw) will reopen and probably request some changes (mainly in the ini what should be default) not today but tomorrow.

[alex47exe]

Sure, any suggestions are welcome! I just copy pasted the custom ini files included in default configuration for generate_emu_config, but forgot to restore the original values for the example ini files.
I guess the whole update should've been split in more commits, but it has been cooking offline for the past 3 weeks or so, without using git to properly commit and document all changes and iterations.

[Detanup01]

:D no worries

[Detanup01]

tools/generate_emu_config/generate_emu_config.py_ini.exe
tools/generate_emu_config/generate_emu_config.*.exe
why?
Also the default plus zips are probably not needed we can let users place steamapi dlls there.
also 7z exists inside the third-party folder

[alex47exe]

___ The exe files are basically just stripped down (no interface) AdvancedRun executables plus their required cfg (commented / documented by me), basically a better method to run various scripts and apps with parameters (including python), with or without interface if required (useful when you need to run an app or script silently -no cmd window flash even-, e.g. when we might need to integrate generate_emu_config into any other automated tool). To be honest, I'm surprised Microsoft doesn't include AdvancedRun by default in an app bundle (and properly integrated with context menu), like they do with Sysinternals Suite or PowerToys, 'cause I really find it the best method to easily run apps and scripts with parameters. Anyway, about those exes:

• *py_ini.exe - only required to run once, installs required python packages to allow running python generate_emu_config.py directly, useful during development
• *py_anon.exe - runs generate_emu_config.py script (needs python installed) in anonymous mode, doesn't fetch achievements, stats and maybe some other things too
• *py_user.exe - runs generate_emu_config.py script (needs python installed) with your own user account and password, to fetch the complete configuration for the game
• *exe_anon.exe (in post_build folder) - runs compiled generate_emu_config in anonymous mode, no need to have an *exe_ini.exe, as all required packages are included in the compiled exe
• *exe_user.exe (in post_build folder) - runs compiled generate_emu_config with your own user account and password, no need to have an *exe_ini.exe, as all required packages are included in the compiled exe

___ If you are referring to steam_api.7z and steam_api64.7z, they include both release and debug versions of the emulator, needed for gse_debug_switch.exe - I think one double-click to run it is a much superior method of changing between release and debug versions when needed, instead of having to manually backup and copy paste them

___ 7za.exe is currently required by gse_debug_switch.exe and gse_acw_helper.exe, and might be required by future tools I currently have some ideas for. I could've used an AutoIt3 UDF script (user defined functions) for zip compression / decompression, but I prefer 7z. Though I used zip compression in generate_emu_config.py for the app_backup folder, screenshots and trailers thumbnails, and parsed download links for SCX scrapper. What I could do is intergrate 7za.exe in a3x script, and unpack it when the script runs, but that would mean every other future a3x script would have to integrate it as well, in the end taking more hdd space than the current method.

[Detanup01]

For 7z:
if you pull eveything from here and even the submodules, you find out the
third-party/deps/linux
third-party/deps/windows
already contains your 7zip.

For the steam apis 7z:
You cannot make push to update because it will always be behind it.

Also probably could make a github action for building the tools rather than having the exe inside there (which I dont like it.)

[alex47exe]

Regarding steam apis 7z, the way I handled them was to manually build the emu on windows, then manually create the steam apis 7z containing both release and debug versions, then push the whole github folder using GitExtensions. Sure, I would love to automate this, but unfortunately I am not experienced with github actions. I guess it would basically mean that the generate_emu_config build would have to be delayed after the emu build, so that the compiled steam apis can be copied and archived to steam apis 7z first.
Same for the already included 7za.exe (though I am not sure why it's an issue) and compiled a3x scripts, both could be achieved automatically through some github actions and / or bat files. So we would need a github action (or modifications to one) to automatically install latest 7zip and AutoIt3, then add 7za.exe to the correct folder, and also build the au3 scripts with Aut2Exe.exe, as explained here: https://www.autoitscript.com/autoit3/docs/intro/compiler.htm
Finally, after all these actions complete successfully, we can build the generate_emu_config executable.

[rcyggdra]

Seems like github issue (I could merge without actually waiting to finish all builds btw) will reopen and probably request some changes (mainly in the ini what should be default) not today but tomorrow.

For familiar people, this PR appears bloated.
If you want to merge this PR, I hope it will be on the new branch instead of replacing the existing one.

[alex47exe]

For familiar people, this PR appears bloated. If you want to merge this PR, I hope it will be on the new branch instead of replacing the existing one.

What do you mean 'bloated'? Sure, maybe it should have been split in more commits, as I said above, but other than that, it only improves generate_config_emu and migrate_gse (though currently it can't generate branches.json, so I don't recommend using it - just regenerate the entire config instead), adds much needed helper tools for automation of various tasks, and corrects spelling mistakes / adds minor changes to wording and punctuation to some readme files. That's literally all it does. It's not 'bloated', it's improved - you just haven't tested it thoroughly to understand it.

[alex47exe]

Just reopened it to be able to fix conflicts with upstream fork, so I can correctly merge the latest changes into my own

[Detanup01]

If you edit the request to be a draft I would be thankfull

[alex47exe]

Sure thing. Haven't really had the time lately to fully and correctly automate all the necessary building and packaging of extra tools I added. There's still some new useful things I'll add into the generate_emu_config tool, like automatically finding and creating shortcuts to the configuration and savegame folders based on data already present in app_details.json (which is part of the reason I decided to include by default some extra json files based on app_details.json) and also scrapped info from PCGamingWiki page for each game - the page link can be obtained automatically through https://www.pcgamingwiki.com/api/appid.php?appid=391220 (example for 'Rise of the Tomb Raider)
I hope that converting the pull request to draft will still let me correct the future differences to be able to correctly merge all future updates to your fork into mine. If it does, thanks, as I was keeping the pull request open for that reason.