Migrating Your Data from Drupal 7 to Drupal 10: Customizing the generated migration

Series Overview & ToC | Previous Article | Next Article - coming July 10th

In the previous article we learned how to generate migrations using the Migrate Upgrade module. As explained, those migrations are generated as configuration entities managed by the Migrate Plus module. Today, we will learn about using migration plugins provided by Drupal Core. After learning about these two methods of handling migrations, we will explain how the example project is set up to perform configuration and content migrations. Finally, we will take a glance at customizing the automatically generated migrations.

Migrate Plus configuration entities

The two primary ways of handling migrations in Drupal 10 are using Migrate Plus configuration entities and Drupal Core's migration plugins. They differ in the filename pattern, the location where the migration files are placed, and how changes to the files are detected.

Let's start with configuration entities. When using this approach, the Migrate Plus module is required. Notice that once the migrations are generated, the Migrate Upgrade module can be uninstalled as the migrations will live in the site's active configuration: the database. When exported, the filename pattern will be migrate_plus.migration.MIGRATON_ID.yml.

It is useful to have all migration related functionality encapsulated in a custom module. When doing so, these migration files are placed inside the config/install folder of the custom module. When the module is enabled for the first time, the migrations will be copied to Drupal's active configuration, which is the database. Further changes to the files inside the module will be ignored unless the configuration is synced again.

This has many implications:

1. The migrations are part of the site's configuration. Doing a configuration export, with drush config:export for example, will create copies of the migrations in the location specified by the $settings['config_sync_directory']. Having two copies of the same file might be confusing. Moreover, neither the version in the config/install folder nor the one in the config_sync_directory folder is the definitive representation of the migration. By default the active configuration is the database, not the exported files. It is possible that the database representation of the migration differs from one or both files.
2. If you need to update the migrations, you need to update the site's configuration. This can get messy quickly. Let's say that you need to update a field setting in a content type. Because of this change, the migration also needs to be updated. You need to be very mindful about the order in which configuration is exported and imported to prevent overwriting either the changes to the content type or the updates to the migration. There are multiple strategies to work around this. If you consider the copy of the migration inside your custom module the canonical version, you can use the drush config:import --partial command to execute a partial configuration import that only affects your custom migrations. You can also use modules like config_devel, config_update, and config_sync to apply updates. In the end, because you are dealing with configuration entities, you can use any workflow to apply configuration changes when the migrations need to be updated.
3. Because the migrations are configuration entities, you can leverage the configuration override system to alter the migrations dynamically. For example, you could use $config['migrate_plus.migration.upgrade_d7_file']['source']['constants']['source_base_path'] = getenv('MIGRATE_D7_FILE_SOURCE'); in your settings.php to use an environment variable to specify the source (domain) to fetch files from in a public files migration. Modules like Configuration Split can also be used to alter the migrations. Again, because you are dealing with configuration entities, any module or workflow that you already implement can be used to manage and alter the migrations.

For reference, below is one way to manage migrations as configuration entities:

# 1) Run the migration.

$ drush migrate:import upgrade_d7_user

# 2) Rollback migration because the expected results were not obtained.

$ drush migrate:rollback upgrade_d7_user

# 3) Change the migration definition file in the "config/install" directory.

# 4) Sync configuration by folder using Drush.

$ drush config:import --partial --source="modules/custom/my_module/config/install"

# 5) Run the migration again.

$ drush migrate:import upgrade_d7_user

Core migration plugins

Another way of handling migrations is using features provided by Drupal core out of the box. That is, using core migration plugins. When using this approach, the filename pattern is MIGRATION_ID.yml and the files are placed inside a migrations folder in your custom module. Rebuilding caches will apply the changes made to the migration files.

In recent years, this has been my preferred approach because making changes to the migration does not require updates to the site's configuration. Also, clearing caches is arguably more straightforward than having to be cautious when syncing updates to migration configuration entities.

For reference, below is one way to manage migrations as core plugins:

# 1) Run the migration.

$ drush migrate:import upgrade_d7_user

# 2) Rollback migration because the expected results were not obtained.

$ drush migrate:rollback upgrade_d7_user

# 3) Change the migration definition file.

# 4) Rebuild caches for changes to be picked up.

$ drush cache:rebuild

# 5) Run the migration again.

$ drush migrate:import upgrade_d7_user

Technical note: Migrations have a dedicated cache bin. During development, you could include the following snippet inside settings.php or in a file included from it to not require cache rebuilds when changes are made.

$settings['container_yamls'][] = DRUPAL_ROOT . '/sites/development.services.yml';

$settings['cache']['bins']['discovery_migration'] = 'cache.backend.null';

Code organization for a custom migration

The example project has three elements that will help organize the code related to the custom migration:

• A custom installation profile: tag1_profile.
• A custom module to store content migrations and related code: tag1_migration.
• A custom module to store configuration migrations and related code: tag1_migration_config.

The custom installation profile is named tag1_profile and it is modeled after the minimal installation profile provided by Drupal core. In our example, sticking with minimal would probably suffice. But in large projects, a custom installation profile can be used to store code related to the site itself. One key assumption of our example project is that the site can be reinstalled at any point using the exported configuration.

While we progress through the examples, we will be migrating some configuration automatically and creating some manually. After every milestone, we will export the configuration. Then, we can use the drush site:install tag1_profile --existing-config command to get a fully configured site on top of which we can run content migrations.

Note: At the time of this writing, a new recipes API was recently introduced to Drupal core in version 10.3.0. Recipes can be used instead of installation profiles to fully configure a site. In our example project, we are sticking with installation profiles. Towards the end of the series, we will look at applying recipes to bring configuration into the site.

The tag1_migration module will contain all the content migrations as core plugins. If necessary, this module can be used to store custom source/process/destination migration plugins, hook implementations, registering event callbacks, and any other code related to migrating content entities.

The tag1_migration_config module will contain all the configuration migrations as core plugins. In fact, this is a submodule of the tag1_migration module. As said before, it is convenient to have all migration related code in one location. Remember that not all configuration will come from a migration. The tag1_migration_config could be removed after the migrated configuration makes its way to the config_sync_directory folder. Instead, I prefer to uninstall the module but keep the code around to have a reference of how some of the configuration was generated via migrations.

From config entities to core migration plugins

The example project will use core migration plugins. However, the generated migrations use configuration entities. Let's learn how to convert one approach to the other using the migration of taxonomy vocabularies.

After following the instructions in the previous article, you should have a file config/migrate_plus.migration.upgrade_d7_taxonomy_vocabulary.yml inside the Drupal 10 installation. The content of the file should look like:

uuid: 79d5ff1c-7924-4508-99b5-ba0ad71d2960
langcode: en
status: true
dependencies: {  }
id: upgrade_d7_taxonomy_vocabulary
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
  - Configuration
migration_group: migrate_drupal_7
label: 'Taxonomy vocabularies'
source:
  plugin: d7_taxonomy_vocabulary
process:
  vid:
    -
      plugin: make_unique_entity_field
      source: machine_name
      entity_type: taxonomy_vocabulary
      field: vid
      length: 30
      migrated: true
  label:
    -
      plugin: get
      source: name
  name:
    -
      plugin: get
      source: name
  description:
    -
      plugin: get
      source: description
  weight:
    -
      plugin: get
      source: weight
destination:
  plugin: 'entity:taxonomy_vocabulary'
migration_dependencies:
  required: {  }
  optional: {  }

We will begin by moving this exported configuration entity into our custom module's migrations folder. At the same time, update the filename to remove the migrate_plus.migration. prefix. Then at the root of the Drupal 10 site, execute this command:

mkdir -p web/modules/custom/tag1_migration/tag1_migration_config/migrations/

mv config/migrate_plus.migration.upgrade_d7_taxonomy_vocabulary.yml web/modules/custom/tag1_migration/tag1_migration_config/migrations/upgrade_d7_taxonomy_vocabulary.yml

Note: Update the command accordingly if you opted out for moving the exported configuration to a ref_migrations folder as discussed in the previous article. What we need is to place the renamed file in the migrations folder of our custom module for configuration migrations.

Now, let's make changes to the file itself:

1. Remove the uuid, langcode, status, and dependencies keys at the start of the file. These are settings related configuration entities. We are switching to using core migration plugins so they are not needed.
2. Remove the field_plugin_method and cck_plugin_method keys if set to null. These settings are used for field-related migrations, but are present in all the generated ones. When set to null they have no effect. Let's keep the migration files tidy by removing them.
3. Add two migration_tags. The first one corresponds to the type of entity being created, or the destination plugin if no entity is created. In this example, taxonomy_vocabulary would be our first custom tag as we are creating taxonomy vocabulary configuration entities. The second tag is either tag1_configuration or tag1_content depending on the type of entity being created. Adding these tags is not strictly necessary, but it helps understand what a file is migrating by scanning its content. Also, we could use these tags to import or rollback multiple migrations at once.
4. Remove the migration_group. Migration groups are a feature provided by the Migrate Plus module to share configuration among multiple migrations. When using Migrate Upgrade, a migrate_drupal_7 group is automatically created and assigned to all the generated migrations. The only shared configuration this group provides is the name of the key to connect to the Drupal 7 database. To the extent possible, I prefer the migration files to be self-contained. That way, there is no need to look in multiple places to know what the migration is doing. We will see some exceptions when dealing with dynamic elements in some migrations. For now, let's remove the migration group.
5. Add a source/key configuration to account for removing the migration group. In our case, the value will be migrate which corresponds to the name of the connection key $databases['migrate] in the settings.migrate.php file. This was explained in article 9.

There are more changes that we could make. Most examples will make changes to the process section of the file. Some will also alter the source and destination sections. Each example will explain what customizations are needed and why we are making them.

For now, the final version of the migration file should look like this:

id: upgrade_d7_taxonomy_vocabulary
class: Drupal\migrate\Plugin\Migration
migration_tags:
  - 'Drupal 7'
  - Configuration
  - taxonomy_vocabulary
  - tag1_configuration
label: 'Taxonomy vocabularies'
source:
  key: migrate
  plugin: d7_taxonomy_vocabulary
process: ... # Redacted for brevity
destination:
  plugin: 'entity:taxonomy_vocabulary'
migration_dependencies:
  required: {  }
  optional: {  }

Today we explored the differences between the two primary ways of handling migrations in Drupal 10 and demonstrated the process of converting migration configuration entities to core migration plugins. In the next article, we will take a deeper look at how the migrations files are written. Specifically, we will examine the syntax and structure of migration files.

Image from Pixabay