The Queue Import module delivers on a very specific need when it comes to large-scale content migrations in Drupal 8 and 9. The gap between the initial migration and deploying the new site to production. Most migration options allow for a single content dump and then the migration process is over--but usually this is where the development process only begins on the new site. So what happens to the content that’s created, edited, or removed during that time if you can’t re-run a migration to update all of your content?
We address the problem with Queue Import. It’s meant to be added to your project, altered for your specific implementation, and then it can be run and re-run with no conflicts so you can import that new content without disrupting either the legacy site or the new site.
Sounds impossible, but on top of that we’re making it disposable. Easy enough to develop quickly and remove entirely once the migration task has been completed.
Implementing Queue Import into your project can be done in the following steps (we derived this process from the Cross-Industry Standard Process for Data Mining or CRISP-DM):
- Understand the business requirements.
- Understand the data
- Prepare the data
- Write the models (content migration scripts, QueueWorkers)
- Evaluate the imported content
- Deploy to production
The short list of adding Queue Import to your project is as follows:
- Run “composer require drupal/queue_import”
- Run “drush pm-enable queue_import”
- Set up database connection information (queue_import.settings)
- Run “drush itest && drush queue-run node_queue_processor” to test functionality
- Start mapping by running “drush d7fm article” where article is the content type from both the legacy site as well as the new D8/9 site
- Rename “MapArticleQueueProcessor.php” in your src/Plugin/QueueWorker directory to “ArticleQueueProcessor.php”
- Set up fields in ArticleQueueProcessor.php (new and legacy fields will be listed automatically)
- Test queueing articles by running “drush d7 article” this will automatically queue up a single article
- Test processing by running “drush queue-run article_queue_processor”
- Evaluate and debug any errors or warnings that arise, and repeat as needed for all content types.
Need to update the existing content? Simple, just re-queue and re-process with any changes either you’ve made to the code or the client has made to the legacy database. The changes will be available upon process completion and you’ll be able to see them live on the site.
The best case scenario is that you can have a full import complete in a matter of weeks and it is then reusable throughout the project’s lifecycle. The basic structure of drush commands creating queue data and then processing the queue data into Drupal entities can be easily leveraged to solve a variety of problems.
It’s best to begin importing content immediately so that the client can see the results and give feedback. The worst possible scenario is that the client waits until the week of deployment and then realizes that a field is incorrect, missing, or isn’t displaying/relating as they had intended. Start the imports as early as possible, re-run regularly, and keep the client updated on the progress.