Make GUI and CLI tools use the same datadir

Currently if a you choose a non-default datadir in the GUI intro screen, the
datadir is ignored by CLI tools. This means `bitcoin-cli` and `bitcoin-wallet`
will try to use the wrong datadir and show errors if they are called without a
-datadir arguments, and `bitcoind` will appear to work but use a different
datadir, not loading the same wallets and settings, and downloading blocks into
the wrong place.

There are also more subtle inconsistencies between GUI and CLI selection of
datadirs such as bitcoin#27273 where GUI might ignore a datadir= line in a
bitcoin.conf that CLI tools would apply.

This PR gets rid of inconsistencies between GUI and CLI tools and makes them
use the same datadir setting by default. It is followup to
bitcoin-core/gui#602 which made GUI and CLI tools use
the same `-dbcache`, `-par`, `-spendzeroconfchange`, `-signer`, `-upnp`,
`-natpmp`, `-listen`, `-server`, `-prune`, `-proxy`, `-onion`, and `-lang`
settings as long as they loaded the same datadir.

The reason for GUI and CLI tools using inconsistent datadirs, is that GUI
stores the datadir path in a `strDataDir` field in
`.config/Bitcoin/Bitcoin-Qt.conf`[^1] which CLI tools ignore. This PR changes
the GUI to instead store the datadir path at the default datadir location
`~/.bitcoin`[^2] as a symlink that CLI tools will already follow, or as a text
file if the filesystem does not support creating symlinks.

If upgrading from a previous version of the GUI and there is only a GUI
datadir, the `strDataDir` setting will be automatically migrated to a symlink
so CLI tools will start using it as well. If CLI and GUI tools are currently
using different default datadirs, the GUI will show a prompt allowing either of
the datadirs to be loaded and optionally set as the common default going
forward.

src/common/init.cpp

@@ -14,7 +14,7 @@
 #include <optional>
 
 namespace common {
-std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn settings_abort_fn)
+std::optional<ConfigError> InitConfig(ArgsManager& args, const fs::path* initial_datadir, SettingsAbortFn settings_abort_fn)
 {
     try {
         if (!CheckDataDirOption(args)) {
@@ -33,6 +33,7 @@ std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn setting
         std::string error;
         fs::path orig_config_path;
         fs::path orig_datadir_path;
+        if (initial_datadir) orig_datadir_path = *initial_datadir;
         if (!args.ReadConfigFiles(error, true, &orig_config_path, &orig_datadir_path)) {
             return ConfigError{ConfigStatus::FAILED, strprintf(_("Error reading configuration file: %s"), error)};
         }
@@ -42,23 +43,9 @@ std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn setting
 
         // Create datadir if it does not exist.
         const auto base_path{args.GetDataDirBase()};
-        if (!fs::exists(base_path)) {
-            // When creating a *new* datadir, also create a "wallets" subdirectory,
-            // whether or not the wallet is enabled now, so if the wallet is enabled
-            // in the future, it will use the "wallets" subdirectory for creating
-            // and listing wallets, rather than the top-level directory where
-            // wallets could be mixed up with other files. For backwards
-            // compatibility, wallet code will use the "wallets" subdirectory only
-            // if it already exists, but never create it itself. There is discussion
-            // in https://github.com/bitcoin/bitcoin/issues/16220 about ways to
-            // change wallet code so it would no longer be necessary to create
-            // "wallets" subdirectories here.
-            fs::create_directories(base_path / "wallets");
-        }
         const auto net_path{args.GetDataDirNet()};
-        if (!fs::exists(net_path)) {
-            fs::create_directories(net_path / "wallets");
-        }
+        if (!CreateDataDir(base_path, error)) return ConfigError{ConfigStatus::FAILED, Untranslated(error)};
+        if (!CreateDataDir(net_path, error)) return ConfigError{ConfigStatus::FAILED, Untranslated(error)};
 
         // Show an error or warning if there is a bitcoin.conf file in the
         // datadir that is being ignored.

src/common/init.h

@@ -5,6 +5,7 @@
 #ifndef BITCOIN_COMMON_INIT_H
 #define BITCOIN_COMMON_INIT_H
 
+#include <fs.h>
 #include <util/translation.h>
 
 #include <functional>
@@ -33,7 +34,7 @@ struct ConfigError {
 using SettingsAbortFn = std::function<bool(const bilingual_str& message, const std::vector<std::string>& details)>;
 
 /* Read config files, and create datadir and settings.json if they don't exist. */
-std::optional<ConfigError> InitConfig(ArgsManager& args, SettingsAbortFn settings_abort_fn = nullptr);
+std::optional<ConfigError> InitConfig(ArgsManager& args, const fs::path* initial_datadir = nullptr, SettingsAbortFn settings_abort_fn = nullptr);
 } // namespace common
 
 #endif // BITCOIN_COMMON_INIT_H

src/qt/bitcoin.cpp

@@ -567,17 +567,18 @@ int GuiMain(int argc, char* argv[])
     /// 5. Now that settings and translations are available, ask user for data directory
     // User language is set up: pick a data directory
     bool did_show_intro = false;
+    fs::path initial_datadir;
     int64_t prune_MiB = 0;  // Intro dialog prune configuration
     // Gracefully exit if the user cancels
-    if (!Intro::showIfNeeded(did_show_intro, prune_MiB)) return EXIT_SUCCESS;
+    if (!Intro::showIfNeeded(did_show_intro, initial_datadir, prune_MiB)) return EXIT_SUCCESS;
 
     /// 6-7. Parse bitcoin.conf, determine network, switch to network specific
     /// options, and create datadir and settings.json.
     // - Do not call gArgs.GetDataDirNet() before this step finishes
     // - Do not call Params() before this step
     // - QSettings() will use the new application name after this, resulting in network-specific settings
     // - Needs to be done before createOptionsModel
-    if (auto error = common::InitConfig(gArgs, ErrorSettingsRead)) {
+    if (auto error = common::InitConfig(gArgs, &initial_datadir, ErrorSettingsRead)) {
         InitError(error->message, error->details);
         if (error->status == common::ConfigStatus::FAILED_WRITE) {
             // Show a custom error message to provide more information in the

src/qt/intro.cpp

@@ -24,6 +24,7 @@
 #include <QMessageBox>
 
 #include <cmath>
+#include <fstream>
 
 /* Check free space asynchronously to prevent hanging the UI thread.
 
@@ -200,22 +201,220 @@ int64_t Intro::getPruneMiB() const
     }
 }
 
-bool Intro::showIfNeeded(bool& did_show_intro, int64_t& prune_MiB)
+// TODO move to common/init
+// TODO write new file before renaming old so less fragile
+// TODO choose better unique filename
+bool SetInitialDataDir(const fs::path& default_datadir, const fs::path& datadir, std::string& error)
 {
-    did_show_intro = false;
+    assert(default_datadir.is_absolute());
+    assert(datadir.is_absolute());
+    const bool link_datadir{datadir == default_datadir};
+    std::error_code ec;
+    fs::file_status status{fs::symlink_status(default_datadir, ec)};
+    if (ec) {
+        error = strprintf("Could not read %s: %s", fs::quoted(fs::PathToString(default_datadir)), ec.message());
+        return false;
+    }
+    if (status.type() != fs::file_type::not_found && (link_datadir || status.type() != fs::file_type::directory)) {
+        fs::path prev_datadir{default_datadir};
+        prev_datadir += strprintf(".%d.bak", GetTime());
+        fs::rename(default_datadir, prev_datadir, ec);
+        if (ec) {
+            error = strprintf("Could not rename %s to %s: %s", fs::quoted(fs::PathToString(default_datadir)), fs::quoted(fs::PathToString(prev_datadir)), ec.message());
+            return false;
+        }
+    }
+    if (link_datadir) {
+        fs::create_directory_symlink(datadir, default_datadir, ec);
+        if (ec) {
+            if (ec != std::errc::operation_not_permitted) {
+                LogPrintf("Could not create symlink to %s at %s: %s", fs::quoted(fs::PathToString(datadir)), fs::quoted(fs::PathToString(default_datadir)), ec.message());
+            }
+            std::ofstream file;
+            file.exceptions(std::ifstream::failbit | std::ifstream::badbit);
+            try {
+                file.open(datadir);
+                file << fs::PathToString(datadir) << std::endl;
+            } catch (std::system_error& e) {
+                ec = e.code();
+            }
+            if (ec) {
+                error = strprintf("Could not write %s to %s: %s", fs::quoted(fs::PathToString(datadir)), fs::quoted(fs::PathToString(default_datadir)), ec.message());
+                return false;
+            }
+        }
+    } else {
+        if (!CreateDataDir(datadir, error)) return false;
+    }
+    return true;
+}
+
+// TODO move low level code out of showIfNeeded to this function
+// TODO move common/init, consolidate arguments/return value
+fs::path GetInitialDataDir(const ArgsManager& args, bool& explicit_datadir, bool& new_datadir, bool& custom_datadir, std::string& error)
+{
+    return {};
+}
 
+bool Intro::showIfNeeded(bool& did_show_intro, fs::path& initial_datadir, int64_t& prune_MiB)
+{
+    // Always show intro if requested.
+    bool show_intro{gArgs.GetBoolArg("-choosedatadir", DEFAULT_CHOOSE_DATADIR) || gArgs.GetBoolArg("-resetguisettings", false)};
+
+    // Check if explicit -datadir command line argument was passed. If it was,
+    // just use the value for the current session and avoid changing the default
+    // datadir that will be used in future sessions. Also avoid showing the
+    // intro dialog if it was not was explicitly requested with -choosedatadir
+    // or -resetguisettings.
+    fs::path datadir{gArgs.GetPathArg("-datadir")};
+    fs::path default_datadir = GetDefaultDataDir();
+    bool explicit_datadir{false}, new_datadir{false}, custom_datadir{false};
     QSettings settings;
-    /* If data directory provided on command line, no need to look at settings
-       or show a picking dialog */
-    if(!gArgs.GetArg("-datadir", "").empty())
-        return true;
-    /* 1) Default data directory for operating system */
-    QString dataDir = GUIUtil::getDefaultDataDirectory();
-    /* 2) Allow QSettings to override default dir */
-    dataDir = settings.value("strDataDir", dataDir).toString();
-
-    if(!fs::exists(GUIUtil::QStringToPath(dataDir)) || gArgs.GetBoolArg("-choosedatadir", DEFAULT_CHOOSE_DATADIR) || settings.value("fReset", false).toBool() || gArgs.GetBoolArg("-resetguisettings", false))
-    {
+    std::error_code ec;
+    if (!datadir.empty()) {
+        explicit_datadir = true;
+    } else {
+        if (settings.value("fReset", false).toBool()) show_intro = true;
+
+        fs::file_status status = fs::symlink_status(default_datadir, ec);
+        if (status.type() == fs::file_type::not_found) {
+            new_datadir = true;
+            datadir = default_datadir;
+        } else if (status.type() == fs::file_type::regular) {
+            std::ifstream file;
+            file.exceptions(std::ifstream::failbit | std::ifstream::badbit);
+            std::string line;
+            try {
+                file.open(datadir);
+                std::getline(file, line);
+            } catch (std::system_error& e) {
+                ec = e.code();
+            }
+            datadir = ec ? default_datadir : fs::PathFromString(line);
+            custom_datadir = true;
+        } else if (status.type() == fs::file_type::symlink) {
+            datadir = fs::read_symlink(default_datadir, ec);
+            custom_datadir = true;
+        } else if (status.type() != fs::file_type::directory) {
+            ec = make_error_code(std::errc::not_a_directory);
+        }
+    }
+
+    // Check if there is a legacy QSettings "strDataDir" setting that should be
+    // migrated.
+    QVariant legacy_datadir_str{settings.value("strDataDir")};
+    bool remove_legacy_setting{false};
+    // TODO consolidate cases below, set remove_legacy_setting in one place
+    if (legacy_datadir_str.isValid()) {
+        fs::path legacy_datadir{fs::PathFromString(legacy_datadir_str.toString().toStdString()).lexically_normal()};
+        if (explicit_datadir) {
+            // If explicit -datadir was passed, let the explicit value take
+            // priority over the legacy value. Discard the legacy value if the
+            // intro dialog is shown and completed, otherwise keep the legacy
+            // value so it can be used when -datadir is not passed.
+            if (show_intro) remove_legacy_setting = true;
+        } else if (legacy_datadir.empty() || legacy_datadir == datadir || legacy_datadir == default_datadir) {
+            // If the legacy datadir string is empty, or the same as the current
+            // datadir, just discard the legacy value.
+            remove_legacy_setting = true;
+        } else if (new_datadir) {
+            // If there is no current datadir, use the legacy datadir.
+            datadir = legacy_datadir;
+            remove_legacy_setting = true;
+            // If showing intro dialog, legacy setting will be shown in the
+            // dialog and saved in the dialog is completed. If not showing
+            // intro, try to save legacy datadir as default now. If it fails to
+            // save, just log a warning. It will still be used this session, and
+            // the legacy setting will be kept so there is a chance to retry the
+            // next session.
+            std::string error;
+            if (show_intro) {
+                remove_legacy_setting = true;
+            } else if (SetInitialDataDir(default_datadir, datadir, error)) {
+                remove_legacy_setting = true;
+            } else {
+                LogPrintf("Warning: failed to set %s as default data directory: %s", fs::quoted(fs::PathToString(datadir)), error);
+            }
+        } else if (show_intro) {
+            // If legacy datadir conflicts with current datadir, but the intro
+            // dialog is going to be shown, just discard the legacy datadir if
+            // the intro dialog is completed. instead of showing an extra dialog
+            // before the intro.
+            remove_legacy_setting = true;
+        } else {
+            // Show a dialog to choose between the legacy and current datadirs.
+            QString gui_datadir{QString::fromStdString(fs::PathToString(legacy_datadir))};
+            QString cli_datadir{QString::fromStdString(fs::PathToString(datadir.empty() ? default_datadir : datadir))};
+            #define Dialog(...)
+            Dialog(R"(
+                The Bitcoin graphical interface (GUI) is configured to use a different default data directory than Bitcoin command line (CLI) tools.
+
+                    GUI default data directory is: {legacy_datadir}
+                    CLI default data directory is: {current_datadir}
+
+                Previous versions of the Bitcoin GUI (24.x and earlier) only used the GUI default directory and ignored the CLI default directory. This version allows choosing which of the two directories to use. It is recommended to set a common default data directory so the GUI and CLI tools such as `bitcoind` `bitcoin-cli` and `bitcoin-wallet` can interoperate and this prompt can be avoided in the future.
+
+                Use GUI data directory and leave defaults unchanged (Same as bitcoin 24.x behavior)
+                Use CLI data directory and leave defaults unchanged
+                Use GUI data directory and set as common default
+                Use CLI data directory and set as common default
+                Choose another data directory and set as default...
+                Quit
+            )");
+            enum {USE_GUI, USE_CLI, SET_GUI_DEFAULT, SET_CLI_DEFAULT, QUIT};
+            if (USE_GUI) {
+                datadir = legacy_datadir;
+                custom_datadir = true;
+                new_datadir = false;
+            } else if (USE_CLI) {
+            } else if (SET_GUI_DEFAULT) {
+                datadir = legacy_datadir;
+                custom_datadir = true;
+                new_datadir = false;
+                std::string error;
+                if (SetInitialDataDir(default_datadir, datadir, error)) {
+                    remove_legacy_setting = true;
+                } else {
+                    LogPrintf("Warning: failed to set %s as default data directory: %s", fs::quoted(fs::PathToString(datadir)), error);
+                }
+            } else if (SET_CLI_DEFAULT) {
+                remove_legacy_setting = true;
+            } else if (QUIT) {
+                return false;
+            }
+        }
+    }
+
+    // If a default or explicit datadir does not exist just show the intro
+    // dialog to confirm it should be created. But if a custom datadir that was
+    // previously selected in the GUI no longer exists, show a dialog to notify
+    // about the problem, since it could happen when an external drive is not
+    // attached, and choosing a new datadirectory would not be desirable.
+    std::string message;
+    if (custom_datadir) {
+        if (datadir.is_absolute()) {
+            fs::file_status status = fs::status(datadir, ec);
+            if (status.type() != fs::file_type::directory) {
+                message = strprintf("Data directory path %s no longer exists or is not a directory", fs::quoted(fs::PathToString(datadir)));
+                if (!ec && status.type() == fs::file_type::not_found) ec = std::make_error_code(std::errc::no_such_file_or_directory);
+                if (ec) message = strprintf("%s: %s", message, ec.message());
+                Dialog(R"(
+                Retry
+                Choose a new data directory location
+                Quit
+               )");
+            }
+        } else {
+            // Error will be displayed in intro dialog
+            ec = std::make_error_code(std::errc::not_a_directory);
+        }
+    }
+
+    did_show_intro = false;
+
+    if (new_datadir) show_intro = true;
+
+    if (show_intro) {
         /* Use selectParams here to guarantee Params() can be used by node interface */
         try {
             SelectParams(gArgs.GetChainName());
@@ -225,8 +424,9 @@ bool Intro::showIfNeeded(bool& did_show_intro, int64_t& prune_MiB)
 
         /* If current default data directory does not exist, let the user choose one */
         Intro intro(nullptr, Params().AssumedBlockchainSize(), Params().AssumedChainStateSize());
-        intro.setDataDirectory(dataDir);
+        intro.setDataDirectory(QString::fromStdString(fs::PathToString(datadir)));
         intro.setWindowIcon(QIcon(":icons/bitcoin"));
+        if (ec) intro.setStatus(FreespaceChecker::ST_ERROR, QString::fromStdString(ec.message()), 0);
         did_show_intro = true;
 
         while(true)
@@ -236,33 +436,34 @@ bool Intro::showIfNeeded(bool& did_show_intro, int64_t& prune_MiB)
                 /* Cancel clicked */
                 return false;
             }
-            dataDir = intro.getDataDirectory();
-            try {
-                if (TryCreateDirectories(GUIUtil::QStringToPath(dataDir))) {
-                    // If a new data directory has been created, make wallets subdirectory too
-                    TryCreateDirectories(GUIUtil::QStringToPath(dataDir) / "wallets");
-                }
-                break;
-            } catch (const fs::filesystem_error&) {
-                QMessageBox::critical(nullptr, PACKAGE_NAME,
-                    tr("Error: Specified data directory \"%1\" cannot be created.").arg(dataDir));
-                /* fall through, back to choosing screen */
+            datadir = fs::PathFromString(intro.getDataDirectory().toStdString());
+            std::string error;
+            if (!datadir.is_absolute()) {
+                intro.setStatus(FreespaceChecker::ST_ERROR, QString::fromStdString("Data directory is not an absolute path."), 0);
+            } else if (!CreateDataDir(datadir, error)) {
+                intro.setStatus(FreespaceChecker::ST_ERROR, QString::fromStdString(strprintf("Could not create data directory: %s", error)), 0);
+            } else if (!SetInitialDataDir(default_datadir, datadir, error)) {
+                intro.setStatus(FreespaceChecker::ST_ERROR, QString::fromStdString(strprintf("Could not set default datadirectory: %s", error)), 0);
+            } else {
+                show_intro = false;
             }
         }
 
         // Additional preferences:
         prune_MiB = intro.getPruneMiB();
-
-        settings.setValue("strDataDir", dataDir);
-        settings.setValue("fReset", false);
-    }
-    /* Only override -datadir if different from the default, to make it possible to
-     * override -datadir in the bitcoin.conf file in the default data directory
-     * (to be consistent with bitcoind behavior)
-     */
-    if(dataDir != GUIUtil::getDefaultDataDirectory()) {
-        gArgs.SoftSetArg("-datadir", fs::PathToString(GUIUtil::QStringToPath(dataDir))); // use OS locale for path setting
     }
+
+    // Save initial datadir so init code can use it to locate bitcoin.conf
+    // (which can point to another datadir). If an explicit -datadir command
+    // line argument was passed and a different datadir was chosen after that in
+    // one of the dialogs dialogs here, call ForceSet to make the dialog value
+    // override the command line argument.
+    initial_datadir = datadir;
+    if (explicit_datadir) gArgs.ForceSetArg("-datadir", PathToString(initial_datadir));
+
+    settings.setValue("fReset", false);
+    if (remove_legacy_setting) settings.remove("strDataDir");
+
     return true;
 }