-
Notifications
You must be signed in to change notification settings - Fork 269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Blueprints] Support Data Liberation importer in the importWxr step #2058
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WIP, Do not merge!
adamziel
changed the title
Blueprints: Use the Data Liberation plumbing in the importWxr step
Blueprints: Use the Data Liberation plugin to import WXR files in the importWxr step
Dec 5, 2024
85 tasks
adamziel
changed the title
Blueprints: Use the Data Liberation plugin to import WXR files in the importWxr step
[Blueprints] Switch importWxr to Data Liberation WXR Importer
Dec 5, 2024
…ose the mapping candidates from the importer and require an explicit decision from the API consumer
importWxr Blueprint step
adamziel
force-pushed
the
import-wxr-via-data-liberation
branch
from
December 7, 2024 19:45
43255d9
to
a5f77d1
Compare
adamziel
changed the title
[Blueprints] Switch importWxr to Data Liberation WXR Importer
[Blueprints] Support Data Liberation importer in the importWxr step
Dec 8, 2024
adamziel
added
[Type] Enhancement
New feature or request
[Feature] Import Export
[Aspect] Data Liberation
labels
Dec 8, 2024
… fix fetch() failures
The failures are now tests that are either flaky or disabled on the trunk. This one seems ready; I'm quite excited! With this PR, we can start solving all the import and export problems Playground was affected by since day 1, e.g. rewriting the URLs in imported content, slow assets downloads, mapping author IDs, etc! |
adamziel
added a commit
that referenced
this pull request
Dec 11, 2024
… offline mode assets Removes data-liberation-core.phar from the list of assets preloaded to support offline mode. It was implicitly added to that list in #2058. The preloading triggers the following error: ``` Failed to execute 'addAll' on 'Cache': Cache.addAll(): duplicate requests ``` The error stops the initialization of the offline mode cache on the first page load. It still work on subsequent page loads. Still, this behavior breaks an E2E test in trunk. Playground doesn't need to preload that asset. It is only required for a psecific flavor of the importWxr step and there's already an established pattern of not preloading plugins, e.g. the SQLite database integration plugin is not preloaded. ## Testing instructions Confirm the Chromium E2E tests pass
adamziel
added a commit
that referenced
this pull request
Dec 11, 2024
… offline mode assets (#2072) Removes data-liberation-core.phar from the list of assets preloaded to support offline mode. It was implicitly added to that list in #2058. The preloading triggers the following error: ``` Failed to execute 'addAll' on 'Cache': Cache.addAll(): duplicate requests ``` The error stops the initialization of the offline mode cache on the first page load. It still work on subsequent page loads. Still, this behavior breaks an E2E test in trunk. Playground doesn't need to preload that asset. It is only required for a psecific flavor of the importWxr step and there's already an established pattern of not preloading plugins, e.g. the SQLite database integration plugin is not preloaded. ## Testing instructions Confirm the Chromium E2E tests pass
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds the Data Liberation WXR importer as an option in the
importWxr
step. The new importer is turned by including the"importer": "data-liberation"
option:When the
importer
option is missing or set to "default," nothing changes in the behavior of the step and it continues using the https://github.com/humanmade/WordPress-Importer importer.The new importer:
This PR is a part of #1894
Implementation details
This
importWxr
step fetches and includes thedata-liberation-core.phar
file. The phar file is built with Box and contains the importer library with its dependencies, which is a subset of the Data Liberation library, a subset of the Blueprints library, and a few vendor libraries.This, unfortunately, means that any changes in the PHP files require rebuilding the .phar file. Here's how you can do it:
You can also build the entire Data Liberation package as a WordPress plugin complete with a wp-admin page:
Both commands will output the built files to
packages/playground/data-liberation/dist
The progress updates are a first-class feature of the new importer. The updated
importer
step receives them in real-time via apost_message_to_js()
call running after every import step. Then, it passes them on to the progress bar UI.Other changes
AsyncHTTP\Client
which deals with raw sockets, Playground's TLS-based network bridge runs the outbound traffic through a cors proxy. Technically,TCPOverFetchWebsocket
gets thecorsProxy
URL passed to theplayground.boot()
call.HTTP/2
header, don't pass duplicateAccess-Control-*
headers, addAccept
to Access-Control-Allow-Headers. FYI @brandonpaytonRemaining work
WP_Stream_Importer
. Exposing this information to the API consumer requires an explicit decision. Do we rewrite it? Or do we ignore it?Follow-up work
WP_Import_Session
API – do we need so many verbosely named methods? Can we achieve the same outcomes with fewer methods?Testing instructions
importWxr
step do, that is it stays at "Importing content" for a moment, fails to fetch media files (CORS issues in network tools), but inserts posts and pages.Related issues