Series Overview & ToC | Previous Article | Next Article
In the previous article we performed an automated migration via the administration interface with tools provided by Drupal core out of the box. This gave us the opportunity to familiarize ourselves with the different modules that assist with the task, settings that affect the migration, and the results when there are no customizations in place.
Today we will leverage the Migrate Plus and Migrate Upgrade modules to generate migrations from the command line, which we can then pick from and customize.
Automated migration using the Migrate Upgrade module
Let’s start by listing the modules that we will use to run the automated migration from the command line:
- Migrate: consists of the core API that can be used to import data from any source.
- Migrate Drupal: allows connecting to a Drupal 6 or 7 site to import configuration and content.
- Migrate Plus: provides extra source, process, and destination plugins; handles migrations as configuration entities; allows reusing of configuration via migration groups; and more. It is a dependency of Migrate Upgrade and a very useful module on its own.
- Migrate Upgrade: provides a Drush command to perform an upgrading from Drupal 6 or 7 to Drupal 10. Optionally, you can generate migrations without executing them.
When using Migrate Drupal UI, we can provide Drupal 7 database credentials via the user interface. However, this time, we will do it via the command line. In either case, we need to add an extra database connection via the $databases array in settings.php or in a file included from it.
In our example project, we have included the second database configuration in settings.ddev.php:
$databases['migrate']['default']['database'] = "db";
$databases['migrate']['default']['username'] = "db";
$databases['migrate']['default']['password'] = "db";
$databases['migrate']['default']['host'] = "ddev-migration-drupal7-db";
$databases['migrate']['default']['port'] = 3306;
$databases['migrate']['default']['driver'] = "mysql";
Refer to the previous article for an explanation of database configurations.
If you followed the instructions in our how-to series up to this point, the Composer command will likely report that there is nothing to install, update or remove. You likely have a partially migrated site. Let’s discard it so we can learn about how to work from the command line. We will install the site again using the minimal installation profile.
To do that, run this command:
cd drupal10
ddev start
ddev composer install
ddev drush --yes -v site:install minimal
ddev launch
ddev drush uli
Now, let’s enable the migration modules:
ddev drush --yes pm:enable migrate migrate_drupal migrate_plus migrate_upgrade
As mentioned in the last article, this will also enable the Password Compatibility (phpass) module. We also discussed how the Migrate API relies on modules being enabled in Drupal 10 to attempt an automated migration. Running the migration from the command line does not change this fact. So, let’s enable the same set of modules that were enabled before the automated migration from the administration interface:
ddev drush --yes pm:enable address announcements_feed automated_cron big_pipe block block_content breakpoint ckeditor5 config contact contextual datetime dblog dynamic_page_cache editor entity_reference_revisions field field_group field_group_migrate field_ui file filter help history image link media media_library menu_link_content menu_ui mysql node options page_cache paragraphs path path_alias pathauto shortcut social_link_field system taxonomy telephone text token toolbar update user views views_ui
Finally, we can trigger the migration from the command line:
ddev drush --yes migrate:upgrade --legacy-db-key='migrate' --legacy-root='http://ddev-migration-drupal7-web'
The --legacy-db-key flag of the migrate:upgrade command expects a database connection key from settings.php pointing to Drupal 7. Per our $databases['migrate'] configuration, the value to use is migrate. The --legacy-root flag is used by the public files migration and it should contain (a) either a fully qualified domain name of the Drupal 7 site to fetch the files from, or (b) a path to a folder in the same server with a copy of the user uploaded files.
Notice there is no flag for private files. This issue aims to add one, but for now you can manually update the private files migration to specify the location of the files. Also, be mindful of other issues related to migrating (private) files.
After running the command you will see a list of executed migrations.
If you were to visit the site now, you would notice that it looks the same as if the migration had been executed from the user interface via the Migrate Drupal UI module. Then, what is the point of doing this from the command line?
For one, it is faster to perform the upgrade this way. It is common to have to run the automated migration multiple times while testing different modules that can help with the upgrade to Drupal 10. But the primary reason is that we can pass an additional flag to the migrate:upgrade command to generate migration files, which we can then pick from and customize.
ddev drush --yes migrate:upgrade --legacy-db-key='migrate' --legacy-root='http://ddev-migration-drupal7-web' --configure-only
Working with migrations is a two step process: (1) write or generate the migration, and (2) execute/run the migration. When the --configure-only flag is not used, the Migrate Upgrade module will perform both actions in one go. When the flag is present, the module will only generate migrations as configuration entities without actually executing them. This will allow us to take the generated migrations and customize them as needed.
The output of the command is similar, but with some subtle differences. Instead of Executed migrations, the command will output Generated migrations. Also, the names of the migrations will be different, prefixed by the string upgrade_. This prefix can be changed by passing the --migration-prefix flag to the migrate:upgrade command.
Exporting generated migrations
If you were to visit the site after running the command with the --configure-only, you will see no configuration or content on the site. If you were using version control, you will notice that no new files are present on the file system. What gives? The command reported that the migrations were generated, but where? The migrations were created in Drupal’s active configuration storage which is the database.
This is one of many examples in which we will see the Migrate API and related modules interacting with other Drupal core APIs. We are not going to get too deep into configuration management for now. Before exporting the generated migrations, let’s define the location where the export should be placed:
$settings['config_sync_directory'] = '../config';
This is added to settings.php or in a file included from it. In our case, we indicated that the export location should be a folder named config at the same level of Drupal’s docroot. Using the tree command inside drupal10 we get the following output:
$ cd drupal10
$ tree -L 1
.
├── composer.json
├── composer.lock
├── config
├── vendor
└── web
The output has been trimmed, but you can see that config and web both exist at the same level under the drupal10 folder at the root of our example repository. Finally, let’s export the configuration:
ddev drush config:export
If you have exported configuration before and there are configuration changes, you will be prompted to confirm the operation. Otherwise, look at the config folder. Those familiar with configuration management will know that exporting configuration this way is an all or nothing operation. That is, you get the configuration for the whole site–not only those related to migrations.
Fortunately, the configuration files follow a pattern that will let us find the generated migrations. In fact, I like to move or copy the migration files to another folder to keep as reference. We will be reinstalling the site a few more times while setting up the project. And later on we will be exporting configuration regularly. Because configuration exports cleans the config sync folder before dumping the new configuration, placing the generated migrations in a different location will let us refer back to them.
cd drupal10
mkdir ref_migrations
cp config/migrate* ref_migrations
ls ref_migrations
We are going to explain the content of those files in a future article but feel free to open them in the text editor of your choice already. For now, a few notes regarding version control. I normally exclude this folder from version control. If using the same name across migration projects, you can define a global .gitignore rule:
touch ~/.gitignore
git config --global core.excludesfile ~/.gitignore
Those commands will create an empty ~/.gitignore and configured it to include ignore rules to apply globally in Git. Edit the file to include ref_migrations/. Alternatively, you can add these files to a new ref-migration branch and even create a pull request from it. The branch can be closed without merging, but being there means you can refer back to the files in the future. Having a pull request is also a good opportunity to invite other team members to review and comment on the generated migrations.
While we will be providing tips regarding version control and configuration management where appropriate, the focus of this series is migrations. You are free to use whatever version control and configuration management strategies you prefer.
In the next article we will explain two different ways of working with migrations: as core plugins and as configure entities. We will also learn how to customize the migrations generated with Migrate Upgrade.
Image by PublicDomainPictures from Pixabay