Native Linux Support #243
Assignees
Labels
A-linux
Area: Linux Support
C-goal
Category: Long term goal. May be code-related or a meta goal.
M-relnotes
Misc: Issue should be listed in the version history for its milestone
Milestone
Comments
Utumno
added
the
C-todo
Utumno
added a commit
that referenced
this issue
Oct 3, 2015
As discussed in #243 vistaDialog is here to stay - so this branch is an effort to at least centralize windows only imports and start cleaning up windows py - windows.py is written in NP style (note for instance the builder pattern in TaskDialog) that really does not help debugging. The need for debugging arose due to #240 - apparently some component (?) of the dialog fails on wine. Pythonize and make it into a package - constants.py, structures.py (comtypes imports) and task_dialog.py. - I am tempted to bin all the TaskDialog API that's not used by Bash - may do if the wine bug keeps eluding me. - _winreg (note the _) we were importing is used to query registry for supported games, user paths and app buttons (...). Moved to bass and from there must be moved to a cross platform module for getting environment parameters (say 'env.py'). A nice old _negative lines count_ refactoring merge.
Utumno
added a commit
that referenced
this issue
Nov 7, 2015
This release features an almost complete refactoring of the codebase, fixing many bugs and opening the way to extend Bash functionality. Bash, being a community work, has over the years become an assortment of hacks, patches and cruft and the program was just about to become non-operable. Example issues - in no particular order (although probably the worst was the horrible performance): - deleting an item in the lists displayed by Bash ran different code depending on whether the user hit delete or right clicked and chose delete - most of those pieces of code were buggy - start up took more than 15 seconds on large Oblivion Load Orders and alt tabbing out and back into Bash would take 6-7 seconds - last one should take no time in the usual case - as seen from the (user visible) debug prints the game detection algorithm run 3-5 times - many, many broken things - including performance in general (most simple operations would hang Bash for some seconds), display glitches and general UI inconsistencies and internal data structures corruption (see #176) Those bugs reflected the state of the code - again in no particular order: - bosh.py (the data model) in 305 was 24635 lines (reduced from 30550 lines in 304.4 - see bosh section below) - basher.py (the UI lists/menus etc) was 18936 lines - the seven tabs that Bash featured were backed up by Tank (2) and List (5) instances - so implementing a feature (say delete) needed edits to at least two classes (usually 7) - the wx library was intermingled with the basher code - the menu items (Links) defined 215 AppendToMenu methods with nearly identical wx code - readability suffered - most attributes/variables were named `data` or `items`, globals hacks crippled the IDE static analysis, copy paste (aka ravioli) code was everywhere. When I started in it I had no idea if refactoring the code was even possible, core Bash is 80k lines of code. It was clear that bosh needed splitting, that candidate number one was the patchers code (well defined and functional) and that basher should follow - but was not at all clear if those were possible, let alone how exactly should the code be structured and how much time would this take. Turns out that it was possible: Bash was indeed coded with modularity in mind (by Wrye ?) but later additions bloated the initial modules beyond recognition and broke the (non enforced) encapsulation - however it wasn't too late. Also turns out that refactoring needed 2 full work years of a single engineer (learning python). The huge performance boost that this release features is the measurable effect of the increase in the code quality. In the items below I try to describe in a quantitative manner what "code quality" means and how it was increased. That's meant to be read from an app that links to the commits - both github and gitk do but additionally github links to the issues and files. ### bosh This release features splitting the patchers' code out of bosh to a dedicated package (whose final structure is still at stake). This reduced bosh to 10516 lines while naturally exposing class dependencies issues, hidden by the "one file program" nature of Bash. The attempt to split bosh had already started in 305, involving: - an effort to eliminate code duplication in C and P patchers using multiple inheritance (chopped off around 1500 lines of pure copy paste code) - see commits in 3c40989 - splitting out what was to become `record_groups.py` - see c3e3543, but don't imitate it - lots has changed since then, including import conventions and commit style - for instance commits on dev that do not launch are strictly forbidden. - splitting out what was to become `parsers.py` - see commits in 08ab6e2 Final (`record_groups.py`, `parsers.py`): 6ae30fc With those out of the way it was finally possible to rip the patchers out of bosh - starting here: 983f259 That was not a mere copy paste rip - warnings were squashed, long lines were wrapped, and most importantly a lot of tricky refactoring was applied to eliminate code duplication - see for ex. commits in: 72fc8a0 for ripping and commits in (note the negative line count): 5d21377, fdab163, 78a85e0 for refactoring See #3 for commit links (finally closed in b6b743b) and #124 for the (still open) package layout issue. Be aware that a lot of things changed since, including import style and the overall package layout - see: e6b2e0e The remaining bosh.py code had some preliminary refactoring applied to it (archives handling, BAIN error handling and refresh etc - see 6ddd4a8) but core bosh refactoring will take place in 307. Major goals are splitting bosh into a number of modules, getting rid of old style classes (DataDict hierarchy, that is the data store singletons) and refactoring settings to efficiently support profiles. ### .data There are 544 occurrences of `.data` in 306 (1081 of `data`) as opposed to 1071 in 305 (2221 `data`) - baring string literals and comments (Pycharm only recently made these scopes available in IDE search, they were much needed). Practically one had to run the debugger to see the type of the object one had at hand, since the variable/attribute name was always 'data': - data is the wrapped dictionary of `DataDict`. Most accesses to this dict were not via the wrappers the DataDict provides. This took many commits to fix - ex 56198be. Before even _thinking_ "performance" read the performance section below (no, none of those modInfos.data.keys() calls was in the middle of a tight loop). - data also was an attribute of the Link hierarchy which had _different meaning based on an isinstance check_: 92e50cd. It could either mean a DataDict subclass (so in Links code there were `self.data.data` accesses) _or_ the list of selected items in the UI, leading to a complete chaos - see the commit above up till the final removal in ffae59e. - UIList subclasses have a `data` attribute that points to the DataDict singletons in bosh. That was dealt with throughout coding 306 as it is closely related to a deeper issue namely the intermingling of UI and model/business logic code. This stems from the 'one file program' nature of Bash so solving this properly means refactoring is at 100%. - to somehow improve readability I introduced idata and pdata attributes for the Installers and People data links - eba53f9 - Those are transitory and meant to also help in eliminating Tank. The links of List subclasses were using self.window.data, whose uses I strived to encapsulate - random example: 0934f1f - all kind of different classes attributes were named `data`. Those required dedicated commits (hey, I couldn't just do a search and replace for "data") - ex. 9839c47. An IDE really helps here. - finally data local variables were renamed - ex. 8e77283 The war with data was going on in parallel with smaller scale wars with `items` (e87dca7 and its parents), `refresh` (4d872ab), etc. ### basher (#163) This is the part of refactoring that I consider almost done - and it was a big pleasure coding it. Since I first tried coding for Bash the dichotomy between balt.Tank and basher.List clearly stood out as a major issue, as did the omnipresence of the wx library and of course the sheer size of basher.py (the UI module). Basher became a package and then the UI API went through a complete rewrite. Basher globals served as a guide in the splitting process, and a metric of class dependencies. Eliminating those globals was an ongoing effort throughout 306 coding - see for instance: 892a19c (screenlist, statusBar), 5852328 (gInstallers), faea771 (modList) till final elimination in bab7c00 Guest star: bosh.links (353be43) Globals served as a link also between Bash's modules (so they evince class and module coupling issues) - eventually encapsulated as static fields of the balt.Link.Frame singleton - see 0690cf3 Link.Frame is an interesting binding of the UI (basher) and the rest of bash that will serve as guide to a later refactoring phase. Especially Link.Frame.modList use in bosh must be eliminated but this is related to the RefreshUI/delete API finalization. The items below detail the UI refactoring. #### wx wx usages went down by an impressive 1148: 305: 2260 usages (balt: 381, basher: 1586) 306: 1112 usages (balt: 418, basher/: 494) Note balt barely going up. This is because, as was stressed in the relevant issue (#190), decoupling the wx library from the code does not simply mean "create wrappers for wx classes/constants and drop them into balt - it means that balt must export an API", so: - balt should not return wx items on which wx methods will be called - see: 9856c69 for example fix. - the code outside of balt should be agnostic of wx ids. Bash code was far form agnostic of wx ids - on the contrary it used them extensively: - balt was exporting it as a function parameter - this mostly resulted in client code calling it with `wx.ID_ANY` (adding wx usages) - see for ex. a555160, 18cf2b1 up till finally dropping the last one of them in 370bef3. - event handlers were using `event.GetId()` - for example fix see 081bfa6. - Link.Execute() has an event parameter which was mostly used with the IdList hack to define lists of menus - this was taken care by the ChoiceLink subclass, introduced in 6fbec32 till finally the last uses of IdList were binned: 000f320 and IdList was removed (967b921) - note that this permitted dropping the `_id` parameter from `Link.__init__`. ItemLink.Execute() `event` parameter is in fact _never used_ in the 306 Link API iteration and will be removed. That's "decoupling wx from Bash code". For commits that combine both fixes see: 46599bb, 9a77bfc The bulk of what remains is to hash up a clever interface for sizers - needs thought as it must accommodate all usages while being powerful enough to cater for new ones. That being said, another thought is incorporating some basher wx dependent stuff back to balt which anyway should be split to a package - at least its bosh depended classes should be clearly separated from its "abstract" wx wrapping API. #### Link API In a gigantic merge (054970e - avoid) the Link subclasses were ripped from basher to per-tab modules - this is not final but it was a necessary step to start splitting basher. As with the rip of patchers this was not a copy paste rip - a new Link class API was introduced. Unlike the patchers rip however (where package structure and the class hierarchy is still at stake) the Link API has few rough edges left in. The guide for designing the Link hierarchy was `AppendToMenu()` calls that featured boilerplate wx code - for wip boilerplate elimination commits see: 6cf40c1 where `EnabledLink` is introduced and 658fcac where my favourite `TransLink` is introduced (AppendToMenu could become quite _complex_ - see also `ChoiceLink`: 6fbec32). The merge chopped off 785 wx uses and AppendToMenu was confined in balt, and reduced to 8 different implementations vs 215 (!) in 305. The Link API significantly evolved since - compare the 306 (semi final) form with the 305 implementation: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/balt.py#L2098-L2114 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/balt.py#L2392-L2446 Previous (old style) Link class explicitly defined _different_ attributes based on an isinstance check, checking if the underlying window was a List or Tank instance. This made the code plain unreadable (given also the names used, see .data section above) while enforcing the Tank/List dichotomy. The isinstance check was finally removed here: 6d260f0. Here is `INI_ListErrors` Link before and after the refactoring: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11168-L11191 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/ini_links.py#L71-L90 Note AppendToMenu code is encapsulated in EnabledLink, wx wrapper introduced for the clipboard, text and help are now class variables, "data" is now "selected" and the lines are wrapped for more readable, declarative and concise code. For a more complicated example see: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11029-L11122 306 :https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/mods_links.py#L84-L161 Previous implementation directly manipulated wx IDs (by global ID_*** complex hacks) while the new one declares classes and lets ChoiceLink superclass take care of creating and appending the menu items. This permitted focusing on the Execute code which as seen got improved while refactored to remove duplication. This class still has rough edges (\_self...) which will be corrected in 307 along with rough edges in the API - for one dropping ItemLink.Execute() `event` parameter. Of note that the Link subclasses featured a lot of duplicate code apart from the wx related stuff - see for instance: 1a9a29e for duplicate code in Mod_Export Links. #### UIList After the Links were ripped from basher it became obvious that refactoring could not progress while balt.Tank and basher.List were both still around. That lead, one fine morning, to a new class - UIList: 1462227. Both Tank and List had an attribute (glist and list (!) respectively) pointing to a ListControl instance (the items displayed in the UI). The first UIList merge: abd5f24 (see commits there for how UIList was engineered from common Tank/List code) has List.list removed in favour of gList: fc7bde8, to be finally encapsulated as _gList in c48ce7d (note there is a completely unrelated `gList` patcher attribute). ##### UIList.SortItems() The first real snug though was unifying the sorting API of Tank and List. ListControl does not support sorting, one has to introduce a hacky dictionary mapping indexes to displayed items' ids (not wx ids) - that's what Tank was doing (undocumented !) while List was using no less than PopulateItems - resulting in bad performance most apparent in the ini tab, where sorting would stat the file system (so clicking on a column header Bash would hang couple seconds). I proceeded unifying the List.SortItems() overrides (6f0adc9), then moving the Tank mechanism to the ListControl class where it belongs (52a4a22) and finally moving SortItems to base: 42d213a - note OnColumnClick callback to base and ignore the isinstance(self, Tank) - corrected later. This made also various other fixups around the Sort API possible, namely adding sort indicators in all Bash lists (only mod tabs had them), fixing master lists sorting (and display) and the beginning of TankData param API deprecation: 7a3b872. ##### UIList.RefreshUI() Second snug and a blocker for fixing performance was centralizing (and cleaning up) the RefreshUI family and underlying PopulateItem(s) (UpdateItem(s) in Tank). Some commits: - Moving RefreshUI to List: 39e1e60 - List.PopulateItem(): f390ab7, ecf30e9 - UIList.PopulateItems(): 350075a - Tank and List `__init__()` eliminated: a510b23 Unifying the RefreshUI API made finally possible to remove List itself: d94a09c. More work is done - for instance see some work on refreshing the details displayed in: 3ff6cf2. ##### UIList.DeleteItems() Once List was no more and Tank a relic of itself it was finally possible to tackle the delete API. First relevant merge is: 02a5891 but the API is yet in alpha due to calling UI code from bosh - see: 6452bcb and its parent merge containing 3da60e6 (the ugly DataDict.delete_Refresh()). This API will hit beta once the data refresh methods return deleted, modified and added - see how this precious info is currently discarded: return bool(_added) or bool(_updated) or bool(_deleted) https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/bosh.py#L3887 ### Performance (#123) As seen by profiling, the performance bottleneck was the libloadorder32.dll we bundled with bash. Given the central place load order operations have in a mod manager application, the introduction of a laggy dll had taken Bash's performance down on its knees since 51f99f2. But this was far from being the only problem. Due to the spaghetti nature of the code (throw some event programming in) Bash was very generous with its refreshes: - it would refresh its data three (3) times on boot - RefreshData callback was triggered whenever Bash took focus - _including_ after Bash itself displayed a dialog to the user - on Oblivion, while Bash had refreshed its mod infos it was _always_ requesting also a refresh from libloadorder, to be sure - so, for instance, when just tabbing out and back in to Bash (with absolutely no changes in the Data dir) libloadorder was triggered and went through its refresh cycle - see ad05f44 for the fix - refresh in BAIN, say, install operations was called twice, along with a double call of modList RefreshUI - refreshing the mods panel would result in also refreshing the saves panel - even when the change that triggered the refresh did not affect saves - see commits in: 63d8dec Now, given that most of those refreshes ended up triggering a refresh from the libloadorder32.dll, Bash would take of the order of 10 seconds for all but the most trivial operations. But even if libloadorder was lightning fast the double and triple refreshes were as much a bug as anything - so the dll lag was even a blessing in disguise as it made apparent the underlying chaos. To solve the dll issue one alternative would be to reimplement it in python. But given that the same dll is used by other mod managers and related apps I anyway had to know what's in there. So I ended up forking libloadorder and going through the C++ ~~pain~~ code: https://github.com/Utumno/libloadorder/ I realized that some of the operations performed by the dll are inherently costly and would actually need a complete rewrite using caches - for instance the mod validity checks (see Utumno/libloadorder#3), which by the way involves yet another library and is not clear what is checked (as is not clear what Bash checks and if those checks are basically performed twice - Utumno/libloadorder#6). As the situation was critical in Oblivion mainly (due to stating the filesystem for mod times) I explicitly bypassed the dll because I anyway have all the data needed to determine the load order - 7cd6533 - while I still use the dll for saving it. Liblo 7.6.0 commit (f5de6c8) apart from performance fixups fixes some other issues with liblo 4 and 6 like writing loadorder.txt when not there, return active plugins in order, adding files to libloadorder.txt that are added to Data, etc - see: Utumno/libloadorder@7bb5cfb But rewriting the dll was not enough - on the Python side of things the load order internal API was a giant hack, complete with event handlers and absolutely no encapsulation, let alone proper caches. I introduced a new module `load_order.py` (for first iteration and design decisions see 98ed549) to centralize load_order handling - will be finalized in 307 but already Plugins is again encapsulated in ModInfos which serves as the basic internal load order API for the rest of Bash. Load order API will be final when implementing features such as "undo load order change" is a breeze, but already makes implementing minor load order related features much easier: 14ecafc. Using the caches I introduced I was able to fix performance of getOrdered() - was O(n^3\*logn) instead of O(n\*logn): 5d7ed0b The double and triple refreshes were a deeper issue - the "one file program" thing - so rewriting refreshes (RefreshUI, the DataDicts refresh, RefreshData etc) - was an ongoing effort throughout 306 development, going hand to hand with the UI refactoring (see #163 for RefreshUI and #123 for links to some of the most relevant commits). In short: - the refreshes on bash dialogs were taken care using a decorator to unbind RefreshData(): a71f3f2 - the BAIN issues were addressed finally here: 8107f7b - BAIN however needs to be split in a dedicated package to start properly fixing it. Goals for the next few releases: - cache ini reads - finalize then profile the load order handling API - profile the refresh of data on boot and optimize it - centralize refresh of data and UI so we can eventually - switch to an event based model of runtime refresh (watchdog ?) ### Warnings Pycharm code inspection emits 2228 warnings in 306 as opposed to 4229 in 305. Metrics are from Pycharm 4.5.4 - using this inspection: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/inspectionProfiles/default_copy_no_typos.xml on this scope: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/scopes/wrye_bash_all_no_cint.xml Warnings come in various sizes and shapes but in general show low code quality. Some I batch fixed (like 'Comparison to None performed with equality operator') but in general I was fixing warnings when visiting nearby code so the remaining ones mostly mark code areas that need attention. On the other end of the spectrum there are warnings that deserve issues of their own (like 'Too broad exception clause' - nicknamed the most diabolical python anti-pattern - see #200). Other interesting cases were: - "Function 'close' doesn't return anything" led to a rewrite of the archives handling code: c29d7b8 - copy paste unused variables were kind of a guide to otherwise heavily undocumented code - but also often pointed to bugs - "method may be static" (264 in 305, went down to 72 in 306) is a classic example of a warning one fixes while editing the code - the remaining ones should be revisited for 307, along with their classes. - unused imports are on their way to extinction - the remaining cases involve `import *` patterns and normalizing the import style, which shall be done when package layout is frozen - and the ugly `__all__` directives one has to update on adding a menu item: 6a2e4a9 - UILists and their links belong together. ### Cruft (#173) Cruft removal got a dedicated issue so returning developers could readily track when and why code they were familiar with got removed. Cruft is not just unused classes, commented out code etc - it is also compatibility with ancient Bash settings (will be removed in 307), non working tabs (like the PM tab also to be removed in 307) and compatibility code with pre 2.7 python (see 6d0a944). One major piece of cruft removed in 306 was BALO: b520591. As you can see removing BALO was needed for tackling refresh - as it was a complex, obsolete piece of code that greatly complicated both refresh and load order handling - and even the Link API (see notoriously complex Mod_BaloGroups(ChoiceLink) subclass here: aa5b89a). ### Formatting & wrapping After some discussion it was clear that PEP8'ing the code would be a pointless effort at the state it was in - however word wrapping is a _functional_ requirement and 79 chars is well thought of, even if you use a wall projector for programming. 305: Total number of non-blank lines: 82463 in 29 files 96.22 lines less than 80 characters (3120 lines exceed) 98.63 lines less than 100 characters (1130 lines exceed) 99.44 lines less than 120 characters (462 lines exceed) 306: Total number of non-blank lines: 85366 in 63 files 97.89 lines less than 80 characters (1803 lines exceed) 99.26 lines less than 100 characters (631 lines exceed) 99.71 lines less than 120 characters (248 lines exceed) Note: - I do not include `cint.py` (and the `chardet` package which should become a submodule) - a big part of the line wrapping was made by manually correcting Pycharm's formatter - so have a close look - the increase of line count is mainly due to 21de440 which adds 5130 lines most (4974) of it being static data in Mopy/bash/game/skyrim.py. Considering there were 34 files added for 34 * 23 = 782 lines of licence text and that wrapping should have produced another 1k lines at least _core Bash code was actually reduced by ~4000 lines_. That is more than welcome and should be imitated - _less code more features_ is a great motto. ### Boot Booting process got a facelift to be finalized in 307. This involves game loading (see 97fc9cf), fixes to the game detection algorithm (e19237c) which used to run 3-5 times on boot, a fix for our custom importer (43e359a) compliments of @mjpieters, and finally a reworking of boot error messages (8dbdd49). ### WINE As reported in #203 "WryeBash v304.2 works beautifully in Wine" - the windows only shell UI stuff broke it though. One of the last things I did for 306 was fixing this - now Bash runs again on WINE: 124f314. Still making it run correctly and in a second phase making Bash run _on bare Linux_ is a code quality goal that got dedicated issues (#240, #241, #243) ------------------------------------------------------------------------ This release is dedicated to Wrye. Special thanks to: - @Sharlikran, @Supierce, @lojack5, @Arthmoor, @alt3rn1ty, @psilocide (aka lefenger), @leandor for testing, support and sharing their knowledge, - @jfpelland, @wduda, @Isenr and @valda for patches, - @mjpieters for saving my sanity at SO a couple times as well as @zed for helping me rewrite the archives handling code. ------------------------------------------------------------------------ # Conflicts: # Mopy/bash/bass.py @@@ -21,9 -21,42 +21,46 @@@ # https://github.com/wrye-bash # # ============================================================================= """This module just stores some data that all modules have to be able to access - without worrying about circular imports.""" + without worrying about circular imports. Currently used to expose layout + and environment issues - do not modify or imitate (ut).""" + import os as _os + import ConfigParser as _cp language = None <<<<<<< HEAD +AppVersion = u"304.4" ======= + AppVersion = u"306" + bashIni = None + + #--Null strings (for default empty byte arrays) + null1 = '\x00' + null2 = null1*2 + null3 = null1*3 + null4 = null1*4 + + def GetBashIni(iniPath=None, reload_=False): ##: needs work + iniPath = iniPath or u'bash.ini' + global bashIni + if reload_ or bashIni is None: + if _os.path.exists(iniPath): + bashIni = _cp.ConfigParser() + bashIni.read(iniPath) + return bashIni + + class Resources: # this belongs to basher but leads to cyclic imports, so... + fonts = None + #--Icon Bundles + bashRed = None + bashBlue = None + bashDocBrowser = None + bashMonkey = None + + # move with its uses to a cross platform 'env.py' module - maybe add bashIni + try: + import _winreg as winreg + except ImportError: + winreg = None >>>>>>> rel-306 ------------------------------------------------------------------------ Build with `2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]` Closes #187.
Utumno
added a commit
that referenced
this issue
Nov 7, 2015
This release features an almost complete refactoring of the codebase, fixing many bugs and opening the way to extend Bash functionality. Bash, being a community work, has over the years become an assortment of hacks, patches and cruft and the program was just about to become non-operable. Example issues - in no particular order (although probably the worst was the horrible performance): - deleting an item in the lists displayed by Bash ran different code depending on whether the user hit delete or right clicked and chose delete - most of those pieces of code were buggy - start up took more than 15 seconds on large Oblivion Load Orders and alt tabbing out and back into Bash would take 6-7 seconds - last one should take no time in the usual case - as seen from the (user visible) debug prints the game detection algorithm run 3-5 times - many, many broken things - including performance in general (most simple operations would hang Bash for some seconds), display glitches and general UI inconsistencies and internal data structures corruption (see #176) Those bugs reflected the state of the code - again in no particular order: - bosh.py (the data model) in 305 was 24635 lines (reduced from 30550 lines in 304.4 - see bosh section below) - basher.py (the UI lists/menus etc) was 18936 lines - the seven tabs that Bash featured were backed up by Tank (2) and List (5) instances - so implementing a feature (say delete) needed edits to at least two classes (usually 7) - the wx library was intermingled with the basher code - the menu items (Links) defined 215 AppendToMenu methods with nearly identical wx code - readability suffered - most attributes/variables were named `data` or `items`, globals hacks crippled the IDE static analysis, copy paste (aka ravioli) code was everywhere. When I started in it I had no idea if refactoring the code was even possible, core Bash is 80k lines of code. It was clear that bosh needed splitting, that candidate number one was the patchers code (well defined and functional) and that basher should follow - but was not at all clear if those were possible, let alone how exactly should the code be structured and how much time would this take. Turns out that it was possible: Bash was indeed coded with modularity in mind (by Wrye ?) but later additions bloated the initial modules beyond recognition and broke the (non enforced) encapsulation - however it wasn't too late. Also turns out that refactoring needed 2 full work years of a single engineer (learning python). The huge performance boost that this release features is the measurable effect of the increase in the code quality. In the items below I try to describe in a quantitative manner what "code quality" means and how it was increased. That's meant to be read from an app that links to the commits - both github and gitk do but additionally github links to the issues and files. This release features splitting the patchers' code out of bosh to a dedicated package (whose final structure is still at stake). This reduced bosh to 10516 lines while naturally exposing class dependencies issues, hidden by the "one file program" nature of Bash. The attempt to split bosh had already started in 305, involving: - an effort to eliminate code duplication in C and P patchers using multiple inheritance (chopped off around 1500 lines of pure copy paste code) - see commits in 3c40989 - splitting out what was to become `record_groups.py` - see c3e3543, but don't imitate it - lots has changed since then, including import conventions and commit style - for instance commits on dev that do not launch are strictly forbidden. - splitting out what was to become `parsers.py` - see commits in 08ab6e2 Final (`record_groups.py`, `parsers.py`): 6ae30fc With those out of the way it was finally possible to rip the patchers out of bosh - starting here: 983f259 That was not a mere copy paste rip - warnings were squashed, long lines were wrapped, and most importantly a lot of tricky refactoring was applied to eliminate code duplication - see for ex. commits in: 72fc8a0 for ripping and commits in (note the negative line count): 5d21377, fdab163, 78a85e0 for refactoring See #3 for commit links (finally closed in b6b743b) and #124 for the (still open) package layout issue. Be aware that a lot of things changed since, including import style and the overall package layout - see: e6b2e0e The remaining bosh.py code had some preliminary refactoring applied to it (archives handling, BAIN error handling and refresh etc - see 6ddd4a8) but core bosh refactoring will take place in 307. Major goals are splitting bosh into a number of modules, getting rid of old style classes (DataDict hierarchy, that is the data store singletons) and refactoring settings to efficiently support profiles. There are 544 occurrences of `.data` in 306 (1081 of `data`) as opposed to 1071 in 305 (2221 `data`) - baring string literals and comments (Pycharm only recently made these scopes available in IDE search, they were much needed). Practically one had to run the debugger to see the type of the object one had at hand, since the variable/attribute name was always 'data': - data is the wrapped dictionary of `DataDict`. Most accesses to this dict were not via the wrappers the DataDict provides. This took many commits to fix - ex 56198be. Before even _thinking_ "performance" read the performance section below (no, none of those modInfos.data.keys() calls was in the middle of a tight loop). - data also was an attribute of the Link hierarchy which had _different meaning based on an isinstance check_: 92e50cd. It could either mean a DataDict subclass (so in Links code there were `self.data.data` accesses) _or_ the list of selected items in the UI, leading to a complete chaos - see the commit above up till the final removal in ffae59e. - UIList subclasses have a `data` attribute that points to the DataDict singletons in bosh. That was dealt with throughout coding 306 as it is closely related to a deeper issue namely the intermingling of UI and model/business logic code. This stems from the 'one file program' nature of Bash so solving this properly means refactoring is at 100%. - to somehow improve readability I introduced idata and pdata attributes for the Installers and People data links - eba53f9 - Those are transitory and meant to also help in eliminating Tank. The links of List subclasses were using self.window.data, whose uses I strived to encapsulate - random example: 0934f1f - all kind of different classes attributes were named `data`. Those required dedicated commits (hey, I couldn't just do a search and replace for "data") - ex. 9839c47. An IDE really helps here. - finally data local variables were renamed - ex. 8e77283 The war with data was going on in parallel with smaller scale wars with `items` (e87dca7 and its parents), `refresh` (4d872ab), etc. This is the part of refactoring that I consider almost done - and it was a big pleasure coding it. Since I first tried coding for Bash the dichotomy between balt.Tank and basher.List clearly stood out as a major issue, as did the omnipresence of the wx library and of course the sheer size of basher.py (the UI module). Basher became a package and then the UI API went through a complete rewrite. Basher globals served as a guide in the splitting process, and a metric of class dependencies. Eliminating those globals was an ongoing effort throughout 306 coding - see for instance: 892a19c (screenlist, statusBar), 5852328 (gInstallers), faea771 (modList) till final elimination in bab7c00 Guest star: bosh.links (353be43) Globals served as a link also between Bash's modules (so they evince class and module coupling issues) - eventually encapsulated as static fields of the balt.Link.Frame singleton - see 0690cf3 Link.Frame is an interesting binding of the UI (basher) and the rest of bash that will serve as guide to a later refactoring phase. Especially Link.Frame.modList use in bosh must be eliminated but this is related to the RefreshUI/delete API finalization. The items below detail the UI refactoring. wx usages went down by an impressive 1148: 305: 2260 usages (balt: 381, basher: 1586) 306: 1112 usages (balt: 418, basher/: 494) Note balt barely going up. This is because, as was stressed in the relevant issue (#190), decoupling the wx library from the code does not simply mean "create wrappers for wx classes/constants and drop them into balt - it means that balt must export an API", so: - balt should not return wx items on which wx methods will be called - see: 9856c69 for example fix. - the code outside of balt should be agnostic of wx ids. Bash code was far form agnostic of wx ids - on the contrary it used them extensively: - balt was exporting it as a function parameter - this mostly resulted in client code calling it with `wx.ID_ANY` (adding wx usages) - see for ex. a555160, 18cf2b1 up till finally dropping the last one of them in 370bef3. - event handlers were using `event.GetId()` - for example fix see 081bfa6. - Link.Execute() has an event parameter which was mostly used with the IdList hack to define lists of menus - this was taken care by the ChoiceLink subclass, introduced in 6fbec32 till finally the last uses of IdList were binned: 000f320 and IdList was removed (967b921) - note that this permitted dropping the `_id` parameter from `Link.__init__`. ItemLink.Execute() `event` parameter is in fact _never used_ in the 306 Link API iteration and will be removed. That's "decoupling wx from Bash code". For commits that combine both fixes see: 46599bb, 9a77bfc The bulk of what remains is to hash up a clever interface for sizers - needs thought as it must accommodate all usages while being powerful enough to cater for new ones. That being said, another thought is incorporating some basher wx dependent stuff back to balt which anyway should be split to a package - at least its bosh depended classes should be clearly separated from its "abstract" wx wrapping API. In a gigantic merge (054970e - avoid) the Link subclasses were ripped from basher to per-tab modules - this is not final but it was a necessary step to start splitting basher. As with the rip of patchers this was not a copy paste rip - a new Link class API was introduced. Unlike the patchers rip however (where package structure and the class hierarchy is still at stake) the Link API has few rough edges left in. The guide for designing the Link hierarchy was `AppendToMenu()` calls that featured boilerplate wx code - for wip boilerplate elimination commits see: 6cf40c1 where `EnabledLink` is introduced and 658fcac where my favourite `TransLink` is introduced (AppendToMenu could become quite _complex_ - see also `ChoiceLink`: 6fbec32). The merge chopped off 785 wx uses and AppendToMenu was confined in balt, and reduced to 8 different implementations vs 215 (!) in 305. The Link API significantly evolved since - compare the 306 (semi final) form with the 305 implementation: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/balt.py#L2098-L2114 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/balt.py#L2392-L2446 Previous (old style) Link class explicitly defined _different_ attributes based on an isinstance check, checking if the underlying window was a List or Tank instance. This made the code plain unreadable (given also the names used, see .data section above) while enforcing the Tank/List dichotomy. The isinstance check was finally removed here: 6d260f0. Here is `INI_ListErrors` Link before and after the refactoring: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11168-L11191 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/ini_links.py#L71-L90 Note AppendToMenu code is encapsulated in EnabledLink, wx wrapper introduced for the clipboard, text and help are now class variables, "data" is now "selected" and the lines are wrapped for more readable, declarative and concise code. For a more complicated example see: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11029-L11122 306 :https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/mods_links.py#L84-L161 Previous implementation directly manipulated wx IDs (by global ID_*** complex hacks) while the new one declares classes and lets ChoiceLink superclass take care of creating and appending the menu items. This permitted focusing on the Execute code which as seen got improved while refactored to remove duplication. This class still has rough edges (\_self...) which will be corrected in 307 along with rough edges in the API - for one dropping ItemLink.Execute() `event` parameter. Of note that the Link subclasses featured a lot of duplicate code apart from the wx related stuff - see for instance: 1a9a29e for duplicate code in Mod_Export Links. After the Links were ripped from basher it became obvious that refactoring could not progress while balt.Tank and basher.List were both still around. That lead, one fine morning, to a new class - UIList: 1462227. Both Tank and List had an attribute (glist and list (!) respectively) pointing to a ListControl instance (the items displayed in the UI). The first UIList merge: abd5f24 (see commits there for how UIList was engineered from common Tank/List code) has List.list removed in favour of gList: fc7bde8, to be finally encapsulated as _gList in c48ce7d (note there is a completely unrelated `gList` patcher attribute). The first real snug though was unifying the sorting API of Tank and List. ListControl does not support sorting, one has to introduce a hacky dictionary mapping indexes to displayed items' ids (not wx ids) - that's what Tank was doing (undocumented !) while List was using no less than PopulateItems - resulting in bad performance most apparent in the ini tab, where sorting would stat the file system (so clicking on a column header Bash would hang couple seconds). I proceeded unifying the List.SortItems() overrides (6f0adc9), then moving the Tank mechanism to the ListControl class where it belongs (52a4a22) and finally moving SortItems to base: 42d213a - note OnColumnClick callback to base and ignore the isinstance(self, Tank) - corrected later. This made also various other fixups around the Sort API possible, namely adding sort indicators in all Bash lists (only mod tabs had them), fixing master lists sorting (and display) and the beginning of TankData param API deprecation: 7a3b872. Second snug and a blocker for fixing performance was centralizing (and cleaning up) the RefreshUI family and underlying PopulateItem(s) (UpdateItem(s) in Tank). Some commits: - Moving RefreshUI to List: 39e1e60 - List.PopulateItem(): f390ab7, ecf30e9 - UIList.PopulateItems(): 350075a - Tank and List `__init__()` eliminated: a510b23 Unifying the RefreshUI API made finally possible to remove List itself: d94a09c. More work is done - for instance see some work on refreshing the details displayed in: 3ff6cf2. Once List was no more and Tank a relic of itself it was finally possible to tackle the delete API. First relevant merge is: 02a5891 but the API is yet in alpha due to calling UI code from bosh - see: 6452bcb and its parent merge containing 3da60e6 (the ugly DataDict.delete_Refresh()). This API will hit beta once the data refresh methods return deleted, modified and added - see how this precious info is currently discarded: return bool(_added) or bool(_updated) or bool(_deleted) https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/bosh.py#L3887 As seen by profiling, the performance bottleneck was the libloadorder32.dll we bundled with bash. Given the central place load order operations have in a mod manager application, the introduction of a laggy dll had taken Bash's performance down on its knees since 51f99f2. But this was far from being the only problem. Due to the spaghetti nature of the code (throw some event programming in) Bash was very generous with its refreshes: - it would refresh its data three (3) times on boot - RefreshData callback was triggered whenever Bash took focus - _including_ after Bash itself displayed a dialog to the user - on Oblivion, while Bash had refreshed its mod infos it was _always_ requesting also a refresh from libloadorder, to be sure - so, for instance, when just tabbing out and back in to Bash (with absolutely no changes in the Data dir) libloadorder was triggered and went through its refresh cycle - see ad05f44 for the fix - refresh in BAIN, say, install operations was called twice, along with a double call of modList RefreshUI - refreshing the mods panel would result in also refreshing the saves panel - even when the change that triggered the refresh did not affect saves - see commits in: 63d8dec Now, given that most of those refreshes ended up triggering a refresh from the libloadorder32.dll, Bash would take of the order of 10 seconds for all but the most trivial operations. But even if libloadorder was lightning fast the double and triple refreshes were as much a bug as anything - so the dll lag was even a blessing in disguise as it made apparent the underlying chaos. To solve the dll issue one alternative would be to reimplement it in python. But given that the same dll is used by other mod managers and related apps I anyway had to know what's in there. So I ended up forking libloadorder and going through the C++ ~~pain~~ code: https://github.com/Utumno/libloadorder/ I realized that some of the operations performed by the dll are inherently costly and would actually need a complete rewrite using caches - for instance the mod validity checks (see Utumno/libloadorder#3), which by the way involves yet another library and is not clear what is checked (as is not clear what Bash checks and if those checks are basically performed twice - Utumno/libloadorder#6). As the situation was critical in Oblivion mainly (due to stating the filesystem for mod times) I explicitly bypassed the dll because I anyway have all the data needed to determine the load order - 7cd6533 - while I still use the dll for saving it. Liblo 7.6.0 commit (f5de6c8) apart from performance fixups fixes some other issues with liblo 4 and 6 like writing loadorder.txt when not there, return active plugins in order, adding files to libloadorder.txt that are added to Data, etc - see: Utumno/libloadorder@7bb5cfb But rewriting the dll was not enough - on the Python side of things the load order internal API was a giant hack, complete with event handlers and absolutely no encapsulation, let alone proper caches. I introduced a new module `load_order.py` (for first iteration and design decisions see 98ed549) to centralize load_order handling - will be finalized in 307 but already Plugins is again encapsulated in ModInfos which serves as the basic internal load order API for the rest of Bash. Load order API will be final when implementing features such as "undo load order change" is a breeze, but already makes implementing minor load order related features much easier: 14ecafc. Using the caches I introduced I was able to fix performance of getOrdered() - was O(n^3\*logn) instead of O(n\*logn): 5d7ed0b The double and triple refreshes were a deeper issue - the "one file program" thing - so rewriting refreshes (RefreshUI, the DataDicts refresh, RefreshData etc) - was an ongoing effort throughout 306 development, going hand to hand with the UI refactoring (see #163 for RefreshUI and #123 for links to some of the most relevant commits). In short: - the refreshes on bash dialogs were taken care using a decorator to unbind RefreshData(): a71f3f2 - the BAIN issues were addressed finally here: 8107f7b - BAIN however needs to be split in a dedicated package to start properly fixing it. Goals for the next few releases: - cache ini reads - finalize then profile the load order handling API - profile the refresh of data on boot and optimize it - centralize refresh of data and UI so we can eventually - switch to an event based model of runtime refresh (watchdog ?) Pycharm code inspection emits 2228 warnings in 306 as opposed to 4229 in 305. Metrics are from Pycharm 4.5.4 - using this inspection: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/inspectionProfiles/default_copy_no_typos.xml on this scope: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/scopes/wrye_bash_all_no_cint.xml Warnings come in various sizes and shapes but in general show low code quality. Some I batch fixed (like 'Comparison to None performed with equality operator') but in general I was fixing warnings when visiting nearby code so the remaining ones mostly mark code areas that need attention. On the other end of the spectrum there are warnings that deserve issues of their own (like 'Too broad exception clause' - nicknamed the most diabolical python anti-pattern - see #200). Other interesting cases were: - "Function 'close' doesn't return anything" led to a rewrite of the archives handling code: c29d7b8 - copy paste unused variables were kind of a guide to otherwise heavily undocumented code - but also often pointed to bugs - "method may be static" (264 in 305, went down to 72 in 306) is a classic example of a warning one fixes while editing the code - the remaining ones should be revisited for 307, along with their classes. - unused imports are on their way to extinction - the remaining cases involve `import *` patterns and normalizing the import style, which shall be done when package layout is frozen - and the ugly `__all__` directives one has to update on adding a menu item: 6a2e4a9 - UILists and their links belong together. Cruft removal got a dedicated issue so returning developers could readily track when and why code they were familiar with got removed. Cruft is not just unused classes, commented out code etc - it is also compatibility with ancient Bash settings (will be removed in 307), non working tabs (like the PM tab also to be removed in 307) and compatibility code with pre 2.7 python (see 6d0a944). One major piece of cruft removed in 306 was BALO: b520591. As you can see removing BALO was needed for tackling refresh - as it was a complex, obsolete piece of code that greatly complicated both refresh and load order handling - and even the Link API (see notoriously complex Mod_BaloGroups(ChoiceLink) subclass here: aa5b89a). After some discussion it was clear that PEP8'ing the code would be a pointless effort at the state it was in - however word wrapping is a _functional_ requirement and 79 chars is well thought of, even if you use a wall projector for programming. 305: Total number of non-blank lines: 82463 in 29 files 96.22 lines less than 80 characters (3120 lines exceed) 98.63 lines less than 100 characters (1130 lines exceed) 99.44 lines less than 120 characters (462 lines exceed) 306: Total number of non-blank lines: 85366 in 63 files 97.89 lines less than 80 characters (1803 lines exceed) 99.26 lines less than 100 characters (631 lines exceed) 99.71 lines less than 120 characters (248 lines exceed) Note: - I do not include `cint.py` (and the `chardet` package which should become a submodule) - a big part of the line wrapping was made by manually correcting Pycharm's formatter - so have a close look - the increase of line count is mainly due to 21de440 which adds 5130 lines most (4974) of it being static data in Mopy/bash/game/skyrim.py. Considering there were 34 files added for 34 * 23 = 782 lines of licence text and that wrapping should have produced another 1k lines at least _core Bash code was actually reduced by ~4000 lines_. That is more than welcome and should be imitated - _less code more features_ is a great motto. Booting process got a facelift to be finalized in 307. This involves game loading (see 97fc9cf), fixes to the game detection algorithm (e19237c) which used to run 3-5 times on boot, a fix for our custom importer (43e359a) compliments of mjpieters, and finally a reworking of boot error messages (8dbdd49). As reported in #203 "WryeBash v304.2 works beautifully in Wine" - the windows only shell UI stuff broke it though. One of the last things I did for 306 was fixing this - now Bash runs again on WINE: 124f314. Still making it run correctly and in a second phase making Bash run _on bare Linux_ is a code quality goal that got dedicated issues (#240, #241, #243) ------------------------------------------------------------------------ This release is dedicated to Wrye. Special thanks to: - Sharlikran, Supierce, lojack5, Arthmoor, alt3rn1ty, psilocide (aka lefenger), leandor for testing, support and sharing their knowledge, - jfpelland, wduda, Isenr and valda for patches, - mjpieters for saving my sanity at SO a couple times as well as zed for helping me rewrite the archives handling code. ------------------------------------------------------------------------ Build with `2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]` Closes #187.
Utumno
added a commit
that referenced
this issue
Dec 23, 2015
Very important refactoring merge, decoupling most of the code from windows specific stuff. Fixes also a regression introduced in 306, namely AccessDeniedError had the wrong code (so would not be caught). Remains win32api - the whole App links stuff needs revisiting but not an immediate priority (most convoluted uses of win32api are in there, plus using WindowsError) as other stuff (like WindowsError) are more important blockers for #243. Especially welcome: moving out of balt the "shell" stuff, centralizing UAC and moving windows stuff out of bosh. Signed-off-by: MrD <the.ubik@gmail.com>
Utumno
referenced
this issue
Mar 21, 2016
Utumno
added a commit
that referenced
this issue
Mar 23, 2016
Now skips top level dirs fomod etc (as opposed to any path that started with those) and only thumbs.db/desktop.ini (add more, like osX ?) files as opposed to any path ending on those (like mydesktop.ini) TODOS - #219: - Still unsure about skipping config... - should we ignore _silentSkipsEnd to in _find_root_index, _refreshBasic ? Using os_sep is under linux #243 - strive for OS independent code. Contains some last minute changes I hope won't introduce a regression this time (as last minute changes on tested commits usually do) - namely bringing os_sep to global scope, which I do not exactly like but I prefer to other alternatives (namely bringing them to local scope inside the methods - see: http://stackoverflow.com/q/36178330/281545 ). Note that in python 2 list comprehension: + os_sep=os_sep, skips_start=tuple( + [x.replace(os_sep, u'') for x in _silentSkipsStart])): would work without moving os_sep to global scope but leaks x and it is not python 3 friendly - I actually initially used a generator and fell on my face: http://stackoverflow.com/q/36184247/281545 I do not also like _refreshBasic but I do not want to expose caching parameters. When I do I signal it by prepending them with _, see size_or_mtime_changed commit (_lstat).
Utumno
added a commit
that referenced
this issue
May 7, 2016
Closes #96 - I used the code from there by Rifter. It is a monkey patch, the bug in the Installer code should be fixed (that is find what locks the Data/ dir in Installer._move, see next commit). env file operations API is still alpha - may need to support globbing after all (see the original code in #96). The issue was again raised here: http://forums.bethsoft.com/topic/1599816-wrye-bash-thread-108/page-6#entry25141206 and explained here: http://forums.bethsoft.com/topic/1599816-wrye-bash-thread-108/page-7#entry25144862 Under #243, #241, #240 too. Signed-off-by: MrD <the.ubik@gmail.com>
Utumno
added a commit
that referenced
this issue
Jun 9, 2016
Untested on linux (a long way to go, see #243) but still a small step. WIP as I may need to rework this for performance and test (os_sep.join vs os.path.join...)
Closed
Utumno
added
C-goal
Utumno
added a commit
that referenced
this issue
Dec 29, 2016
Note I only try to extract the asset from the first bsa that seems to have it - I will wait for feedback on this. Note also that I now extract all assets at once - I need better error messages. I dropped the existence check in extra_bsas - a KeyError is much faster (provided bsaInfos is refreshed) This makes libbsa redundant (and it's ctype wrapper) - under #243 The patch builds a tiny bit faster for games with string files but more accurate profiling is needed.
Utumno
added a commit
that referenced
this issue
May 5, 2017
Small fixup in BAIN - missed return value when changed install to return a dict in ba2c62b Icons fixups from @nycz for linux - under #243 - Some of the icon names were converted to lowercase in the code to match their real file names. - the AddIconFromFile had to go, since it tried to load images before wx had initialized fully which made everything crash Utumno: it curiously worked on windows - however, AddIconFromFile was a hasty API I introduced in 08de8ad and I am more than happy to see it go. Mopy/bash/env.py: - Pruning some unneeded GPath calls - not needed to append a null, added comment as I keep forgetting this, see: https://github.com/tjguk/winshell/blob/1509d211ab3403dd1cff6113e4e13462d6dec35b/winshell.py#L236
Utumno
pushed a commit
that referenced
this issue
May 28, 2017
When running on Linux: - uses the os 7z executable instead of the bundled 7z.exe - tries to find the path to the Java executable with the shell command "command -v java" instead of using Windows paths - drag and drop is disabled since it works really inconsistently on wxGTK - wx.TextCtrl.HideNativeCaret() is disabled since it's not implemented at all in wxGTK - the mbcs encoding is only available on Windows - win32api.GetFileVersionInfo is replaced by a native reimplementation of it that works on Linux, in the new module linux.py - the UAC test is skipped I also bypassed a path check that threw an exception by default for Linux paths and fixed a Windows-only hardcoded path. Since paths are still case insensitive among other things, there's a long way to go before it works fully on Linux (see #243) but this should at least let it start without crashing.
|
i've been thinking about the problem with case (in)sensitive paths a bit since that is the by far biggest hurdle to get bash running on linux. the only decent option i can come up with is generating and keeping a dict of lower-case-to-mixed-case path pairs that bash's internal path system deals with. basically this:
i have no idea if this is as slow as it sounds, but it should be fairly easily cached. also, i'm thinking this should only be done this way on case-sensitive file systems so hopefully windows shouldn't be affected. one fairly significant problem would of course be case-sensitive file conflicts, for example if someone manually adds the directory |
And that's where I stopped reasoning about that What I think is a better approach is to refactor "bash's internal path system" till it's absolutely clear where the fine line between the path handling and the system calls limit lies - pathlib has separate mixins for abstract paths and concrete paths and Pitrou certainly though deep about that ;) So help needed in refactoring - I filled up an issue ( #368 ) and my current wip branch contains a new data structure that is very related to the problem (LowerDict) - a very tricky merge I have not found the time to push to dev. Keep in mind that it took me couple of years to finally realize what's the right way to address that (simpler) problem - there are many details that I can't really convey but LowerDict is the tool for that job. Keep also in mind that refactoring in Path class led to a dramatic decrease in memory: https://bethesda.net/community/post/371559 So this is not an easy problem. Bottom line we should revisit this once we've switched to pathlib - and that's way, way ahead. Meanwhile there's a lot to be done. |
|
first of all i hope i didn't come across as arrogant or sounding like i had the silver bullet solution or anything D: i know things are a lot more complicated than they might seem for someone not intimately familiar with the source code and i don't expect this to be solved easily or quickly. this is, however, more or less the one thing that keeps bash from working on linux so the sooner it's done (with or without my help) the sooner i can use (and by extension develop) bash properly on my own computer so i'm eager to do what i can to help, specifically with this. with regards to your branch and LowerDict, is that so tightly tied to future Path changes that it's not meaningful for me to start diving deep into Path system and pathlib yet? |
Haha, not at all, same here
Preferably with your help - dunno of anyone else that actually uses Bash on pure linux :P
It's certainly meaningful. I really don't know how to guide you though - the hypothesis that paths are case insensitive is implicit in the code. I think you should start with the hypothesis that case insensitive paths are indeed unique (that is ignore cases like Data/Textures and Data/textures) and investigate if all works - tests... Bash's APIs are case aware however (both Path and LowerDict) - that's a good thing I will merge Asap but meanwhile, having a look in there, helping (once merged) to clearly name attributes that use case insensitive data structures (prepend But also probably tackling other better defined refactoring issues will make you more familiar with the code (and help with current dev) - for instance here's one I would like someone to take off my plate: #372 |
Utumno
pushed a commit
that referenced
this issue
Jun 4, 2017
When running on Linux: - uses the os 7z executable instead of the bundled 7z.exe - tries to find the path to the Java executable with the shell command "command -v java" instead of using Windows paths - drag and drop is disabled since it works really inconsistently on wxGTK - wx.TextCtrl.HideNativeCaret() is disabled since it's not implemented at all in wxGTK - the mbcs encoding is only available on Windows - made GetFileVersionInfo os agnostic. The new env.get_file_version still uses win32api if available - Encapsulate the last of winreg (getting the path to BOSS.exe). All winreg stuff is now in env.py or windows.py. - the UAC test is skipped I also bypassed a path check that threw an exception by default for Linux paths and fixed a Windows-only hardcoded path. Since paths are still case insensitive among other things, there's a long way to go before it works fully on Linux (see #243) but this should at least let it start without crashing.
nycz
added a commit
that referenced
this issue
Jun 29, 2017
They should be replaced by either calls to Bash's own path system, calls to os.path.join, or at least os.sep to make them (more) os agnostic. The code isn't super pretty because I mostly went with a one-to-one simple replace instead of refactoring it more thoroughly. This is merely a basic cleanup to make Bash more runnable on Linux while hopefully not blowing up on Windows in the process. Needless to say, this isn't ready for prime time quite yet. Still untouched are stuff in the game/constants.py files and some things either Windows-specific or where I suspected it might be some kind of internal paths instead of proper file system paths. Under #243
Infernio
added a commit
that referenced
this issue
May 28, 2023
I wanted this release to be smaller for once. 500 commits and nearly a year later, mission failed :( Hopefully we'll manage to keep the next one smaller. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525, #650) The first major change in 311 was the beginning of the road towards proper Fallout 4 records support and hence a fully function Bashed Patch in FO4. - 00c9ee8 implemented a key piece of the records infrastructure, NVNM. It also dropped some hacks to further pave the way. The first two FO4 record types were implemented. - dc6311a refactored CTDA and implemented the next seven record types. - 641bc73 tackled the single biggest challenge of the entire undertaking: VMAD. As a bonus, if you enjoy rants about stupid design decisions, see c5b5d8a. The next ten record types were also implemented. - 28578a6 moved some record types that are identical between Skyrim and FO4 to brec and implemented the next eighteen records. - This was immediately followed by bb7e5ae, which slightly refactored GameInfo.init() but mostly just implemented the next twelve record types. - 3d8b23e implemented nine more record types. - 9bfb6de implemented two more. Also, @BeermotorWB began the porting process by bringing Tweak Settings to FO4 in 57b2443. There are about sixty FO4 record types left to implement. This will be concluded in 312. I chose this part for the title of the release ("Fallout 4 preparation") since it's what occupied my brain during most of 311's development. ----------------------------------------------------------------------- ### Python 3 (#618, #619, #644) Since moving to Python 3 in 310 (see #460), we've been taking full advantage of everything py3 offers us. Some example commits: - 8de5f40 cleaned up one of the most annoying files to edit, common_subrecords.py. - cc198c9 shaved hundreds of lines off by replacing verbose nested OrderedDict instantiations with regular, now-ordered dicts. - fd1d4d9 took advantage of ordered dicts to drop duplication between the records list we used to keep track of records in a Mob* class and the id_records dict we used for fast random access by FormID. Now we only need the dict. - In bbb8fa2, @lojack5 used Python 3's type annotations to refactor our flags handling. We also upgraded to Python 3.10 in 7b3573c and to Python 3.11 in 69e7e3a. We also optimized a bunch of central code for e.g. the BP, FName and Path to take advantage of changes like 3.11's adaptive interpreter, which gave pretty good speedups. We've also been doing what we call 'fdusting' - replacing any random pieces of code that look like a + '.esp' with f-strings, i.e. f'{a}.esp'. Another (made-up) example: 'Loaded %d images' % len(images) becomes f'Loaded {len(images)} images'. This has been happening as a sort of "if you're editing in the vicinity, fix any you touch" thing, so spread out over tons of commits. F-strings are one of the best things about py3 :) ----------------------------------------------------------------------- ### Records (#480, #647) The eternal #480 rears its head again, as it does every release. First off, every single merge mentioned in the FO4 section above also falls under this section. In addition to that, we are approaching setDefault. Uprooting that will give us major speed boosts, simplify code, fix lots of random "the BP is editing this value for some reason" bugs (all of those are due to defaults), etc. One major blocker was FormID handling (see section "FormIDs (#637)" below), now cleaned up. Some other commits related to records refactoring in 311: - 28e0734 reworked defaults handling, making defaults explicit where they are needed and eliminating them from most structures in the records code. - fd1d4d9, already mentioned in the "Python 3 (#618, #619, #644)" section above, reduced duplication in the record_groups code by getting rid of the 'self.records' list in favor of just the (thanks to py3, now ordered) 'self.id_records' dict. - 71520f7 massively reduced code duplication in the various games' init() methods and the __slots__ definitions for all the record types for a total of nearly 1000 LOC removed! - bbb8fa2 by @lojack5, already mentioned in the "Python 3 (#618, #619, #644)" section above, took advantage of Python 3's type hints to refactor our flags handling. The biggest merge of this development cycle also falls in this category. In a truly herculean merge (d2152aa), @Utumno tackled record_groups, turning it from a TODO-riddled mess that didn't really know if it wanted to be Oblivion-only or game-agnostic into a modern API that will serve us well for the future (e.g. for FO4, which makes QUST a new top-level group). Seriously, this refactoring was so complex, its merge has two sub-merges. ----------------------------------------------------------------------- ### FormIDs (#637) One of the major refactoring goals of 311 was tackling FormIDs. We used to repesent them as either integers (called "short FormIDs") or tuples of strings and integers (called "long FormIDs"). Note that the latter became tuples of FNames and integers in 310 (see section "FName (#543)" in the 310 release commit. This was achieved in 4964710, which is once again an entire branch squashed down to a commit so as to not break dev. They are now stored as a class instead (brec.FormId). This required careful engineering and testing to make sure we don't kill BP performance. For the full story, see the linked commit. ----------------------------------------------------------------------- ### Localization (#55, #500) We're trying to replace all '%s' specifiers with more useful '%(something)s' specifiers. This is still nowhere close to being finished, but 2442285 began the process. Once this is done, we'll want to rethink how we allow people to contribute localizations - see #500 for more info. ----------------------------------------------------------------------- ### Epic Games Store & GOG (#646, #648) No release is complete without adding support for new games, and 311 is no exception. Bethesda released most of their games on GOG and some on the Epic Games Store, so we added support. See the linked issues and the referencing commits for more information. The biggest difficulty was adding support for yet *another* method of game detection (for the Epic Games Store). Still to come is refactoring the game detection on the Steam side, since some of these rereleases will actually overwrite the registry key used by the Steam versions, making it impossible for WB to reliably detect the installation of both versions. This will also pave the way for Proton-based game detection on Linux (see "Linux (#243)" section below). ----------------------------------------------------------------------- ### Nexus integration (#459) 5e688fd prepared us for proper Nexus integration, which will be one of the big feature goals of 312, by merging a nexus.py file that wraps the Nexus API into the source tree so that keeping it up to date is easier. ----------------------------------------------------------------------- ### FileInfos (#336) Some refactoring work on FileInfos and related APIs occurred as well, mostly on the INIs. See 22c8c5e, 2495972 and b3e4116 for details. ----------------------------------------------------------------------- ### wx (#190) That's right, more of this. - 6f2bd1d finally de-wx's the mighty UIList (which backs all of WB's tabs) by making it inherit from our PanelWin instead of wx.Panel. Turns out it was that easy (I'm sure it had nothing to do with the metric tons of refactoring that had been done towards this in 310, 309, 308 and 307). - f7dd4dc decouples us from wx with regards to the default, builtin art from wx we used to use. We instead now use SVGs (mostly from Font Awesome - don't worry, I was careful with the licensing), which also gives us some consistency across platforms. - d1290a5 begins the fight against balt.ListBoxes, an absolutely horrible class that not only had a really weird API but also featured terrible usability. I wrote a mini-novel on this refactoring, see the linked commit. The second part arrived later, in 15444ca. - 6659eb3 features some good old de-wx'ing of dialogs for opening/saving/choosing files. Also included is some refactoring of status bar-related classes. ----------------------------------------------------------------------- ### Usability (#625, #643, #645, #652, #656) Back on my bullshit, aka trying to make Wrye Bash more usable. For 311, we have: - bfdee3e, which added a Ctrl+S shortcut (and matching link for discoverability) to make frequent saving easier. - 47ef378, which aimed to improve the INI Edits tab's usability by making it more consistent with the other tabs (e.g. adding Alt+Click support, opening tweaks via Enter or double click, etc.). - effcf83, which added links for LO undo/redo to improve their discoverability. - 486a640, which restricted the red background color to timestamp conflicts since it kept confusing people. See the relevant issue (#656) for some more background info on this change. - 941fb25, which was born out of me going "hey, I've never used the Fit Contents and Fit Header options before, let's try that", really liking how much easier it made adjusting columns for Master lists, then noticing that it applied *globally* and so nuked my carefully adjusted columns on the Mods tab. Instead, each setting now only applies to the tab/list you enable it on, allowing you to use it for just the lists where it helps you. - d1290a5, which was already mentioned in "wx (#190)". ListBoxes was just no good for usability. They didn't remember sizes, always started out being sized wrong and don't even wrap text correctly when you do resize them. Replacing them with custom classes allowed us to massively improve the usability of all kinds of popups and dialogs throughout WB. The second part of this refactoring arrived later, in 15444ca. ----------------------------------------------------------------------- ### Image handling (#366, #557) As an offshoot of #190, we have the venerable #366. Image handling is still very much a work in progress, but f7dd4dc brings us a lot closer. Fleshing out the image API and making it ready for SVGs, it also replaces many of our somewhat crusty images with SVGs that can scale to any size we need. It also further decouples us from wx (see the second entry in the "wx (#190)" section up above). ----------------------------------------------------------------------- ### File operations (#241) In b5bb441, @lojack5 changed our file operations backend on Windows to use the newer IFileOperation API instead of the old SHFileOperation API. This was no easy task, as pywin32's support for IFileOperation is broken for some reason and so @lojack5 had to write a wrapper from scratch. ----------------------------------------------------------------------- ### Update checking (#663) Come 312, we want to replace our nine(!) Nexus page with a single one in the Modding Tools section. The LOOT team recently took the same step. The main reason is simply because updating nine pages for every release *sucks*. Even moreso if the release gets detected as a virus. However, the problem with moving to a single page is discoverability. The Nexus team have indicated that they're interested in adding the ability for pages in the Modding Tools section to indicate that they support certain games, so that they'll show up in the search results for searches on those games' pages, but that's not implemented yet. To that end, we implemented an update check. Once 312 is released on the new Nexus page, we can stop updating the other pages and direct users to the new one via the update popup. Of course, the update check can be disabled in the settings and a manual version of the check is also accessible from there. You can also adjust how frequently the update check happens (the default is once every hour). ----------------------------------------------------------------------- ### Linux (#243) Near the end of 311's development, I bought an NVME SSD and took the opportunity to redo my dual boot environment. After backing up and restoring the Windows part of my dual boot, Windows promptly decided to ruin my day by first breaking Firefox and Thunderbird, then breaking the entire Start menu. Since I can't really get work done when I have to use stupid workarounds like "Windows+R > explorer.exe or wt.exe > launch the app I actually want to launch", I said "screw it" and moved entirely to Linux. Born out of that is the last part of 311: - 1d6db12 fixed a smorgasbord of random problems when using WB on Linux. - 1fe6d2d made BAIN wizard and FOMOD images work on Linux. - 1581953 fixed the GTK webview refusing to load when we have a file with '&' in its filename. - 1d07fc3 implemented sending files to the recycle bin on Linux. Most of these apply to our (even more WIP) macOS support as well. ----------------------------------------------------------------------- There were many, many, *many* other changes - again, 500 commits. One person who hasn't been mentioned so far is @sibir-ine, who contributed lots of minor improvements to WB's out-of-the-box usability (e.g. default BAIN directories) and tirelessly helped with monitoring the Discord server and recording the reported bugs on our project board. Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @sibir-ine, @lojack5, @BeermotorWB and many more that GitHub's contribution tracker doesn't list.
Infernio
added a commit
that referenced
this issue
May 28, 2023
I wanted this release to be smaller for once. 500 commits and nearly a year later, mission failed :( Hopefully we'll manage to keep the next one smaller. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- The first major change in 311 was the beginning of the road towards proper Fallout 4 records support and hence a fully function Bashed Patch in FO4. - 00c9ee8 implemented a key piece of the records infrastructure, NVNM. It also dropped some hacks to further pave the way. The first two FO4 record types were implemented. - dc6311a refactored CTDA and implemented the next seven record types. - 641bc73 tackled the single biggest challenge of the entire undertaking: VMAD. As a bonus, if you enjoy rants about stupid design decisions, see c5b5d8a. The next ten record types were also implemented. - 28578a6 moved some record types that are identical between Skyrim and FO4 to brec and implemented the next eighteen records. - This was immediately followed by bb7e5ae, which slightly refactored GameInfo.init() but mostly just implemented the next twelve record types. - 3d8b23e implemented nine more record types. - 9bfb6de implemented two more. Also, @BeermotorWB began the porting process by bringing Tweak Settings to FO4 in 57b2443. There are about sixty FO4 record types left to implement. This will be concluded in 312. I chose this part for the title of the release ("Fallout 4 preparation") since it's what occupied my brain during most of 311's development. ----------------------------------------------------------------------- Since moving to Python 3 in 310 (see #460), we've been taking full advantage of everything py3 offers us. Some example commits: - 8de5f40 cleaned up one of the most annoying files to edit, common_subrecords.py. - cc198c9 shaved hundreds of lines off by replacing verbose nested OrderedDict instantiations with regular, now-ordered dicts. - fd1d4d9 took advantage of ordered dicts to drop duplication between the records list we used to keep track of records in a Mob* class and the id_records dict we used for fast random access by FormID. Now we only need the dict. - In bbb8fa2, @lojack5 used Python 3's type annotations to refactor our flags handling. We also upgraded to Python 3.10 in 7b3573c and to Python 3.11 in 69e7e3a. We also optimized a bunch of central code for e.g. the BP, FName and Path to take advantage of changes like 3.11's adaptive interpreter, which gave pretty good speedups. We've also been doing what we call 'fdusting' - replacing any random pieces of code that look like a + '.esp' with f-strings, i.e. f'{a}.esp'. Another (made-up) example: 'Loaded %d images' % len(images) becomes f'Loaded {len(images)} images'. This has been happening as a sort of "if you're editing in the vicinity, fix any you touch" thing, so spread out over tons of commits. F-strings are one of the best things about py3 :) ----------------------------------------------------------------------- The eternal #480 rears its head again, as it does every release. First off, every single merge mentioned in the FO4 section above also falls under this section. In addition to that, we are approaching setDefault. Uprooting that will give us major speed boosts, simplify code, fix lots of random "the BP is editing this value for some reason" bugs (all of those are due to defaults), etc. One major blocker was FormID handling (see section "FormIDs (#637)" below), now cleaned up. Some other commits related to records refactoring in 311: - 28e0734 reworked defaults handling, making defaults explicit where they are needed and eliminating them from most structures in the records code. - fd1d4d9, already mentioned in the "Python 3 (#618, #619, #644)" section above, reduced duplication in the record_groups code by getting rid of the 'self.records' list in favor of just the (thanks to py3, now ordered) 'self.id_records' dict. - 71520f7 massively reduced code duplication in the various games' init() methods and the __slots__ definitions for all the record types for a total of nearly 1000 LOC removed! - bbb8fa2 by @lojack5, already mentioned in the "Python 3 (#618, #619, #644)" section above, took advantage of Python 3's type hints to refactor our flags handling. The biggest merge of this development cycle also falls in this category. In a truly herculean merge (d2152aa), @Utumno tackled record_groups, turning it from a TODO-riddled mess that didn't really know if it wanted to be Oblivion-only or game-agnostic into a modern API that will serve us well for the future (e.g. for FO4, which makes QUST a new top-level group). Seriously, this refactoring was so complex, its merge has two sub-merges. ----------------------------------------------------------------------- One of the major refactoring goals of 311 was tackling FormIDs. We used to repesent them as either integers (called "short FormIDs") or tuples of strings and integers (called "long FormIDs"). Note that the latter became tuples of FNames and integers in 310 (see section "FName (#543)" in the 310 release commit. This was achieved in 4964710, which is once again an entire branch squashed down to a commit so as to not break dev. They are now stored as a class instead (brec.FormId). This required careful engineering and testing to make sure we don't kill BP performance. For the full story, see the linked commit. ----------------------------------------------------------------------- We're trying to replace all '%s' specifiers with more useful '%(something)s' specifiers. This is still nowhere close to being finished, but 2442285 began the process. Once this is done, we'll want to rethink how we allow people to contribute localizations - see #500 for more info. ----------------------------------------------------------------------- No release is complete without adding support for new games, and 311 is no exception. Bethesda released most of their games on GOG and some on the Epic Games Store, so we added support. See the linked issues and the referencing commits for more information. The biggest difficulty was adding support for yet *another* method of game detection (for the Epic Games Store). Still to come is refactoring the game detection on the Steam side, since some of these rereleases will actually overwrite the registry key used by the Steam versions, making it impossible for WB to reliably detect the installation of both versions. This will also pave the way for Proton-based game detection on Linux (see "Linux (#243)" section below). ----------------------------------------------------------------------- 5e688fd prepared us for proper Nexus integration, which will be one of the big feature goals of 312, by merging a nexus.py file that wraps the Nexus API into the source tree so that keeping it up to date is easier. ----------------------------------------------------------------------- Some refactoring work on FileInfos and related APIs occurred as well, mostly on the INIs. See 22c8c5e, 2495972 and b3e4116 for details. ----------------------------------------------------------------------- That's right, more of this. - 6f2bd1d finally de-wx's the mighty UIList (which backs all of WB's tabs) by making it inherit from our PanelWin instead of wx.Panel. Turns out it was that easy (I'm sure it had nothing to do with the metric tons of refactoring that had been done towards this in 310, 309, 308 and 307). - f7dd4dc decouples us from wx with regards to the default, builtin art from wx we used to use. We instead now use SVGs (mostly from Font Awesome - don't worry, I was careful with the licensing), which also gives us some consistency across platforms. - d1290a5 begins the fight against balt.ListBoxes, an absolutely horrible class that not only had a really weird API but also featured terrible usability. I wrote a mini-novel on this refactoring, see the linked commit. The second part arrived later, in 15444ca. - 6659eb3 features some good old de-wx'ing of dialogs for opening/saving/choosing files. Also included is some refactoring of status bar-related classes. ----------------------------------------------------------------------- Back on my bullshit, aka trying to make Wrye Bash more usable. For 311, we have: - bfdee3e, which added a Ctrl+S shortcut (and matching link for discoverability) to make frequent saving easier. - 47ef378, which aimed to improve the INI Edits tab's usability by making it more consistent with the other tabs (e.g. adding Alt+Click support, opening tweaks via Enter or double click, etc.). - effcf83, which added links for LO undo/redo to improve their discoverability. - 486a640, which restricted the red background color to timestamp conflicts since it kept confusing people. See the relevant issue (#656) for some more background info on this change. - 941fb25, which was born out of me going "hey, I've never used the Fit Contents and Fit Header options before, let's try that", really liking how much easier it made adjusting columns for Master lists, then noticing that it applied *globally* and so nuked my carefully adjusted columns on the Mods tab. Instead, each setting now only applies to the tab/list you enable it on, allowing you to use it for just the lists where it helps you. - d1290a5, which was already mentioned in "wx (#190)". ListBoxes was just no good for usability. They didn't remember sizes, always started out being sized wrong and don't even wrap text correctly when you do resize them. Replacing them with custom classes allowed us to massively improve the usability of all kinds of popups and dialogs throughout WB. The second part of this refactoring arrived later, in 15444ca. ----------------------------------------------------------------------- As an offshoot of #190, we have the venerable #366. Image handling is still very much a work in progress, but f7dd4dc brings us a lot closer. Fleshing out the image API and making it ready for SVGs, it also replaces many of our somewhat crusty images with SVGs that can scale to any size we need. It also further decouples us from wx (see the second entry in the "wx (#190)" section up above). ----------------------------------------------------------------------- In b5bb441, @lojack5 changed our file operations backend on Windows to use the newer IFileOperation API instead of the old SHFileOperation API. This was no easy task, as pywin32's support for IFileOperation is broken for some reason and so @lojack5 had to write a wrapper from scratch. ----------------------------------------------------------------------- Come 312, we want to replace our nine(!) Nexus page with a single one in the Modding Tools section. The LOOT team recently took the same step. The main reason is simply because updating nine pages for every release *sucks*. Even moreso if the release gets detected as a virus. However, the problem with moving to a single page is discoverability. The Nexus team have indicated that they're interested in adding the ability for pages in the Modding Tools section to indicate that they support certain games, so that they'll show up in the search results for searches on those games' pages, but that's not implemented yet. To that end, we implemented an update check. Once 312 is released on the new Nexus page, we can stop updating the other pages and direct users to the new one via the update popup. Of course, the update check can be disabled in the settings and a manual version of the check is also accessible from there. You can also adjust how frequently the update check happens (the default is once every hour). ----------------------------------------------------------------------- Near the end of 311's development, I bought an NVME SSD and took the opportunity to redo my dual boot environment. After backing up and restoring the Windows part of my dual boot, Windows promptly decided to ruin my day by first breaking Firefox and Thunderbird, then breaking the entire Start menu. Since I can't really get work done when I have to use stupid workarounds like "Windows+R > explorer.exe or wt.exe > launch the app I actually want to launch", I said "screw it" and moved entirely to Linux. Born out of that is the last part of 311: - 1d6db12 fixed a smorgasbord of random problems when using WB on Linux. - 1fe6d2d made BAIN wizard and FOMOD images work on Linux. - 1581953 fixed the GTK webview refusing to load when we have a file with '&' in its filename. - 1d07fc3 implemented sending files to the recycle bin on Linux. Most of these apply to our (even more WIP) macOS support as well. ----------------------------------------------------------------------- There were many, many, *many* other changes - again, 500 commits. One person who hasn't been mentioned so far is @sibir-ine, who contributed lots of minor improvements to WB's out-of-the-box usability (e.g. default BAIN directories) and tirelessly helped with monitoring the Discord server and recording the reported bugs on our project board. Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @sibir-ine, @lojack5, @BeermotorWB and many more that GitHub's contribution tracker doesn't list.
1 task
Infernio
added a commit
that referenced
this issue
Jul 19, 2023
A step towards making WB's data/settings files more cross-platform.
E.g. if you copy a Windows .dat file over to Linux (or vice versa), then
WB can now 'fix' doc browser paths as long as they were pointing to
files in the Data folder. If they weren't, we simply ignore them. Much
better than the current behavior, which will create either a file with a
ridiculously long name ('G:\steam\steamapps\...\Cobl.html') or a ton of
nested folders ('G:', then 'steam', then 'steamapps', etc.) in the CWD.
Mopy/bash/bolt.py:
Notice me being stupid and forgetting to actually convert the separators
for the _s here, back when I added convert_separators.
Under #243
Infernio
added a commit
that referenced
this issue
Jul 19, 2023
Our Link.Frame hackery clearly doesn't work on wxGTK, these would just do nothing there. That said, this might break things on Windows or in some other contexts I didn't test. Under #243 requirements.txt: scripts/helpers/github_wrapper.py: Bump requirements Yay, we can drop the custom wxPython build! One parameter we use in github_wrapper was deprecated in the newest PyGithub version, so move to the recommended replacement. scripts/build.py: build: Bump NSIS version to 3.09 Nothing that should affect us in this release.
Infernio
added a commit
that referenced
this issue
Jul 19, 2023
Based on the AUR package. Not sure if any other distros package it and what paths they would use, but should be easy enough to adjust once we get some feedback. Under #243
Infernio
added a commit
that referenced
this issue
Aug 19, 2023
By abandoning the use of the locale-specific '%c' formatting specifier. Might lose us some locale-specific niceness, but Python on Linux errors when trying to strptime one of them (and it probably isn't a good idea to rely on locale behavior right now, what with our set_c_locale nonsense still in place). Under #243
Infernio
added a commit
that referenced
this issue
Sep 25, 2023
Now avoids the registry. This has several advantages:
- Works on Linux
- Works with Starfield, which does not have a custom registry key
anymore
- Works with Morrowind, which is *supposed* to have a custom registry
key, but I've never managed to generate one on Windows
- No more need to launch the game once for WB to detect it (and no more
need to relaunch it after verifying)
Note that this isn't enough for proper automatic game detection on Linux
yet, because we don't set the local AppData folder correctly (should set
itself to automatically point to the compatdata protonprefix when the
Steam version is detected).
Note also the new dependency - didn't want to hack together my own
parser when there's a perfectly good, *tested* parser available. Could
have made it optional, but I chose not to for two reasons:
1. It's pure Python, which means it's highly unlikely to ever block us
from upgrading to a new Python version.
2. Detecting Steam games is one of the most important things for WB to
be able to do. If someone does not have PyMuPDF installed, that's
not a big deal - they just won't be able to view PDFs. But if
someone didn't have vdf installed, that would mean WB would just
quietly ignore all their Steam games. Steam is the No. 1 platform
for Bethesda games on PC, so I think it's very important that this
work out of the box (including on all OSes).
Under #243
Infernio
added a commit
that referenced
this issue
Oct 10, 2023
A whole bunch of Linux fixes and enhancements, making Wrye Bash quite functional on Linux (as long as you mount your Data folder case-insensitively, which we now detect and warn about). Other improvements and fixes include: - WB will now detect installed Proton games and automatically set the personal/LocalAppData path for them to use the right protonprefix. - WB will now warn about case-sensitively-mounted Data folders and warn the user while also offering advice on how to make it case-insensitive. Such init warnings now also show an option to exit WB entirely instead of continuing. - Fixed a race condition (at least on btrfs), which caused BAIN installations that had to create a new folder to fail. - Fixed default temp folder detection using the Data folder itself for determining what filesystem the temp dir should be placed on rather than the *contents* of the Data folder. - Fixed clearReadOnly missing some quotes in its 'find' command, which was causing errors in stderr when the path to the Data folder contained spaces (note most games necessarily have this - e.g. "steamapps/common/Fallout 4"). However, this is WIP because now the chmod itself fails for me. Not sure why we even need this. - Fixed case sensitivity handling for strings extracted from BSAs on Linux, making localized plugins like the CC content in SSE/FO4 work properly in WB on Linux. - Added support for Ctrl+Tab to cycle through WB's tabs. On Windows we get this natively, on Linux we have to implement it ourselves. Under #243
Infernio
added a commit
that referenced
this issue
Oct 21, 2023
This merge further improves our Linux support and also gets us closer to closing #241 by attacking holes in our file operation backend APIs. Another FOMOD implementation hole was closed as a side effect (installing one file to multiple destinations, a thing that FOMODs are of course 100% capable of doing, because they weren't complicated enough yet). Don't get fooled by the diffstat, a new dependency and the inclusion of the license (plus the forgotten license of packaging) bloats the diff counter. Git tip of the day: $ git diff HEAD^ --shortstat -- . ':(exclude)Mopy/LICENSE-THIRD-PARTY' 13 files changed, 403 insertions(+), 286 deletions(-) There's the true diffstat, without the bloat :) Under #241 and #243
Infernio
added
the
M-relnotes
|
I'm gonna mark this as done for 312. Linux support is close to 100% usable now, only some QOL stuff and the tool launchers left (which are WIP anyways, what with #570). |
|
Still need to mark it as supported by removing the need for |
Infernio
added a commit
that referenced
this issue
Dec 29, 2023
This makes OMOD extraction work on Linux, but I don't know if getting rid of the .head there is actually correct or if it'll just break something else. And while we're at it, does this need a _retry too? Under #243
Infernio
added a commit
that referenced
this issue
Jan 4, 2024
Move 'run' code to env; make get_file_version work on Linux Moves run_exe to env wrappers and makes it return a Popen reference we can access from within app_buttons or wherever else a subprocess is called. Reimplements get_file_version to use native binary reading utilities on Linux. It Just Works! Refines Java detection a bit. Added some type hints and comments. Infernio: Some minor followup fixes for Linux - Prefer JAVA_HOME if set - Skip 0.0.0.0 and 1.0.0.0 versions to match the Windows version - Fix reading of xSE versions (uses commas and spaces instead of dots) - Get rid of u'' prefixes and use f-strings Deprint about missing 'strings' program and add a cache It's not *that* slow (adds up to about 0.02s on boot on my Linux system), but there's no reason to keep reading the same files over and over just in case the version changed. Added a cache decorator to the Windows version too, since the same consideration applies there, the pywin32 calls are cheap, but not *that* cheap. Utumno: ExeLauncher is used for .exe files with the exception of LOOT - best thing here is to forget about special handling of LOOT as one can always add the unix path to bash.ini. So I removed this particularly nasty override - whenever you see an os. check outside env is nasty - and I moved the _run_exe code for non .exe files to linux.AppLauncher.launch_app. Untested but should work - move allow_create to base and disallow handling of .lnk files in linux. This is the base implementation, we can drop return False now that linux is fleshed out a bit :) xdg-open reported not to take any args with or without the '--' - nit in get_file_version -> avoid looping twice, hopefully 'FileVersion' and co come early. Under #243 Co-authored-by: Infernio <infernio@icloud.com> Co-authored-by: MrD <the.ubik@gmail.com>
Infernio
added a commit
that referenced
this issue
Jan 4, 2024
On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.
Infernio
added a commit
that referenced
this issue
Jan 4, 2024
On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bash should be able to run on linux. Opening this to start investigating.
Case sensitive filesystem - oopsie woopsie, Bash has absolutely no notion of that - mostly not a problem anymore since we require the Data folder to be case-insensitive (see a8942b4). Various bits in WB still assume case though, like
barbOur file operation backend needs to do a better job of supporting Linux (and non-Windows platforms in general) - see File operations #241
Proton awareness, so that most users won't need to - worked on in 20dd955, done in ae12a5c
app buttons code is tightly coupled with
_winregwin32api- Linux does not really use app buttons code anymore, but that just points to us needing to rethink these entirely (Rethink app launchers #570)hunt down all hardcoded backslashes (duh) - ref 2e65a93 - tentatively marking this done
move winreg from bass and to an environment detection module - cross platform
WindowsError, duh:http://stackoverflow.com/q/1210118/281545, http://stackoverflow.com/a/8890190/281545, http://stackoverflow.com/a/2678797/281545, http://stackoverflow.com/a/22027543/281545 -> edit done see Replace WindowsError with OSError #252this (traceback solved in a975191 but check also warns) :
Of course the biggest hurdle is the dlls... If it proves easy we can compile them to.sos so much the betterAll DLLs dropped by rewriting relevant code in Python:
Blockers:
Future work:
/usror/opt, we need profilesThe text was updated successfully, but these errors were encountered: