Series Overview & ToC | Previous Article | Next Article

This step-by-step series has covered a lot of ground on planning and preparing for a Drupal 7 to Drupal 10 migration. Today, we start putting that knowledge into practice by automatically migrating content types. First, we will execute a migration to pull all content types in Drupal 7. Then, we will customize the migration to remove unnecessary content types. Finally, we will learn how a separate node title label migration also affects content type configuration in Drupal 10.

Prerequisites

For this section, we are going to switch to the start branch in the repository. This will contain the basic project set up for the Drupal 7 and Drupal 10 sites, but no migrations. You can always refer to the main branch for the final version of the examples. If you have not done so, clone the repository and switch to the start branch.

git clone https://github.com/tag1consulting/d7_to_d10_migration.git d7_to_d10_migration
git checkout start

Moving forward, we will be executing migrations to migrate data from Drupal 7 into Drupal 10., This process requires both environments to be running. Refer to Article 8 -Example repository setup and Drupal 7 site audit - for an explanation on how to set up the Drupal 7 site and Article 9 - Performing an automated migration - for an explanation on how to set up the Drupal 10 site.

For reference, use the commands below to get both sites up and running.

For Drupal 7:

# From the root of the repository.
cd drupal7
ddev start
# See article 7 for importing the database and files.
ddev launch
ddev drush uli

For Drupal 10:

# From the root of the repository.
cd drupal10
ddev start
ddev composer install
# Install the site from existing configuration
ddev composer si
ddev launch
ddev drush uli

The ddev composer si is a custom Composer script that drops the current Drupal 10 database and installs the site from the existing configuration. Run this command before proceeding with today's example. Once the site is installed, there is no need to execute this command again unless otherwise instructed. The custom script is defined in the root composer.json file as such:

"scripts": {
    "si": "drush --yes sql:drop && drush --yes -v site:install tag1_profile --existing-config"
},

Migrating content types

In Drupal 10, content types are configuration entities. Back in Article 1- Getting to Know Your Migration, we mentioned that migrating configuration automatically makes sense when there are little or no configuration changes compared to Drupal 7. Our example will include a significant number of content model changes. To account for that, we could create all the configuration manually and only automate the migration of content. Instead, we will cherry-pick and customize the automated migration of a subset of Drupal 7's configuration. After automating part of the process, we will manually create or update configuration in Drupal 10 to reflect the desired content model.

We learned how to generate automatic migrations in article 10, and we talked about code organization for a custom migration project in article 11.As mentioned previously, we will use a tag1_migration_config module to store configuration migrations and related code. To do so, copy the upgrade_d7_node_type migration from the reference folder into our custom module and rebuild caches for the migration to be detected.

cd drupal10
cp ref_migrations/migrate_plus.migration.upgrade_d7_node_type.yml web/modules/custom/tag1_migration/tag1_migration_config/migrations/upgrade_d7_node_type.yml
ddev drush cache:rebuild

Note that while copying the file, we also changed its name and placed it in a migrations folder inside our custom module. This was discussed in Article 11 - Customizing the generated migration as part of the changes to convert a migration configuration entity to a Core migration plugin. After copying the file, make the following changes:

  • Remove the following keys: uuid, langcode, status, dependencies, field_plugin_method, cck_plugin_method, and migration_group.
  • Add two migration tags: node_type and tag1_configuration.
  • Add key: migrate under the source section.

For an explanation of these changes, refer back to article 11. After the modifications, the file should look like this:

id: upgrade_d7_node_type
class: Drupal\migrate\Plugin\Migration
migration_tags:
  - 'Drupal 7'
  - Configuration
  - node_type
  - tag1_configuration
label: 'Node type configuration'
source:
  key: migrate
  plugin: d7_node_type
  constants:
    preview: 1
process:
  type:
    -
      plugin: get
      source: type
  name:
    -
      plugin: get
      source: name
  description:
    -
      plugin: get
      source: description
  help:
    -
      plugin: get
      source: help
  title_label:
    -
      plugin: get
      source: title_label
  preview_mode:
    -
      plugin: get
      source: constants/preview
  display_submitted:
    -
      plugin: get
      source: display_submitted
  new_revision:
    -
      plugin: get
      source: options/revision
  create_body:
    -
      plugin: get
      source: create_body
  create_body_label:
    -
      plugin: get
      source: body_label
  third_party_settings/menu_ui/available_menus:
    -
      plugin: static_map
      bypass: true
      source: available_menus
      map:
        main-menu: main
        management: admin
        navigation: tools
        user-menu: account
  third_party_settings/menu_ui/parent:
    -
      plugin: static_map
      bypass: true
      source: parent
      map:
        'main-menu:0:': 'main:'
        'management:0:': 'admin:'
        'navigation:0:': 'tools:'
        'user-menu:0:': 'account:'
destination:
  plugin: 'entity:node_type'
migration_dependencies:
  required: {  }
  optional: {  }

Executing the content type migration

Drupal migrations are a two step process. You first write the migration plugin and then you execute it. As discussed in Article 11 - Customizing the generated migration, migrations can be written as Drupal Core plugins or as configuration entities using Migrate Plus. As for executing them, there are multiple migration runners. The most commonly used are Drush and the Migrate Tools module. Both options provide commands to execute migrations from the command line.

Migrate Tools also provides a user interface to run migration configuration entities from a web interface. See this Migrate Tools issue discussing support for Core plugins in the administration interface.

Technical note: The commands provided by Drush core and Migrate Plus are mostly the same. There are flags that are only present in one or the other. See this issue to learn about some differences between the two.

Migrate Tools provides extra functionality like shared configuration, which we will not implement. We do not need a web interface for running migrations and that feature does not currently support Core migration plugins. Therefore, we will use Drush to run the migrations from the command line throughout the whole series.

Execute the following commands to make sure our content type migration is being detected properly.

cd drupal10
ddev drush migrate:status upgrade_d7_node_type

You should see that there are eight records to process. Those correspond to the eight content types present in Drupal 7. You can see them at https://migration-drupal7.ddev.site/admin/structure/types

If the command fails saying Invalid migration IDs: upgrade_d7_node_type, make sure that the tag1_migration_config module is enabled and that the upgrade_d7_node_type.yml file is placed in the migrations folder inside the module. The full location within the repository should be drupal10/web/modules/custom/tag1_migration/tag1_migration_config/migrations/upgrade_d7_node_type.yml

If the command does not return any output other than table headers, it means it is not possible to connect to the Drupal 7 database. Make sure the Drupal 7 DDEV site is running and that a connection string named migrate exists in settings.ddev.php. See article 9 - Performing an Automated Migration - for details on setting up a database connection string.

Now, let's execute the migration. Moving forward, it is assumed that the commands are executed within the drupal10 folder at the root of the example project repository.

ddev drush migrate:import upgrade_d7_node_type

You should see that eight items were processed. The report should also indicate that eight items were created—content types in this case. You can see the migrated content types at https://migration-drupal10.ddev.site/admin/structure/types

Customizing the content type migration

In Article 8 - Example repository setup and Drupal 7 site audit, we drafted a lightweight migration plan. We specified that three content types from Drupal 7 would not be migrated to Drupal 10:

  • Nodes of type speaker will be migrated as user entities.
  • Nodes of type sponsor will be migrated as taxonomy term entities.
  • Nodes of type swag will not be migrated and the content type will be dropped altogether.

These content model changes are also reflected in the source site audit document discussed in Article 3 - Source Site Audit - An In depth Analysis and populated in Article 8 - Example repository setup and Drupal 7 site audit.

Now we need to update the migration to prevent the three content types listed above from being migrated. Identify the following lines in the upgrade_d7_node_type migration:

process:
  type:
    -
      plugin: get
      source: type

And change them to:

process:
  # @modified
  type:
    -
      plugin: skip_on_value
      source: type
      value:
        - speaker
        - sponsor
        - swag
      method: row
      message: 'Content type not needed in new site.'

As discussed before, the migrate API will fetch and process one record at a time. In this case, we are processing content types. For each Drupal 7 content type, the d7_node_type source plugin will populate the type source property with the machine name of the content type.

Using the skip_on_value process plugin, we prevent content types from being imported if their machine name matches one of the array elements defined in the value configuration option. We skip the whole record from being processed by setting method: row. Additionally, we set a custom message to explain the reason why the content type was skipped.

It is important to know what process plugins are provided by Drupal core and contributed modules like Migrate Plus. There are many operations that can be performed without writing custom code. During the series, we explain those relevant to customizing the migrations in the example project , but we recommend researching what is available on your own. Refer to the 31 days of migration series for examples and explanations on the use of various process plugins. We also encourage you to review the previous article, which explains in more detail the syntax of migration files and the process pipeline.

If we were to run the migrate:status command now, we would see that eight records had been imported and that there are 0 unprocessed records. If we were to run the migrate:import command, you would see that no items are processed. Instead of doing either, execute the following commands:

# Rollback the migration. Removes all previously imported content types.
ddev drush migrate:rollback upgrade_d7_node_type

# Rebuild caches. This makes changes to the migration file to be detected.
ddev drush cache:rebuild

# Import the migration again. This time, only a subset of the content types will be migrated.
ddev drush migrate:import upgrade_d7_node_type

# Check the messages related to this migration.
ddev drush migrate:messages upgrade_d7_node_type

# Check the status of this migration.
ddev drush migrate:status upgrade_d7_node_type

Let’s break down these commands.

First, we rollback the upgrade_d7_node_type migration. This removes the content types that were previously imported. Rolling back a migration does not always result in deleting the imported entities. For example, when reverting a node translation, the primary entity might remain in the system. Also, the deletion of entities can be queued to be performed at a later time. We will highlight the cases when rolling back a migration does not result in the immediate deletion of the entity.

Because we are using Drupal core migration plugins, we need to rebuild caches for our changes to the file to take effect. Alternatively, during development we could set the discovery_migration cache bin to cache.backend.null. This was discussed in Article 11 - Customizing the generated migration as a way to bypass the need to rebuild caches. Throughout the series, we will be explicit as to when the migration cache is cleared and will rebuild caches manually.

Then we execute the migration again using the migrate:import command. This time you will see that out of the eight processed items, five were created and three were ignored. With the migrate:messages command we can check that records with Source ID(s) speaker, sponsor, and swag have no Destination ID(s). This means that those records were not imported into Drupal 10. The Message column should contain a familiar message: "Content type not needed in new site."

Console message

Technical note: The Level column in the migrate:messages command is populated by a call to \Drupal\migrate\MigrateException::getLevel(). Possible values are constants in the \Drupal\migrate\Plugin\MigrationInterface whose name starts with MESSAGE_. An error is stored as 1, a warning as 2, a notice as 3, and an informational message as 4.

Finally, we execute migrate:status to verify there are no pending records to process. It is possible to restrict how many records to import by using the --limit flag of the migrate:import command or by dynamically altering the migration query to add a range. Unless you are manually limiting the number of records to process, you should see 0 unprocessed records.

A number other than 0 usually indicates a problem somewhere in the migration:

  • If the number is positive, it is possible that the source plugin's configured ID columns are not unique.
  • Another option for a positive value is that the column used as a highwater property is not unique.
  • If negative, it usually means that records in Drupal 7 have been deleted since the last time the migration was executed. In this case, deleting a content type in Drupal 7 after we execute this migration would result in a negative number.

At this point, there should be 5 content types in the Drupal 10 site. Except for the body field attached to the Article (article) and Basic page (page), no other field has been added to any content types. Migrating fields require four different migrations which will be discussed in future articles.

Those paying very close attention will notice there is a setting in the Venue content type in Drupal 10 that does not match the corresponding Drupal 7 value. That is, the label of the title field. In Drupal 7, it is Name while in Drupal 10 it is Title.

Migrating node title labels

That brings us to migrating node title labels. There is a separate migration for importing the label for node titles: upgrade_d7_node_title_label. Copy it from the reference migrations folder into our custom module and rebuild caches for the new migration to be detected:

cd drupal10
cp ref_migrations/migrate_plus.migration.upgrade_d7_node_title_label.yml web/modules/custom/tag1_migration/tag1_migration_config/migrations/upgrade_d7_node_title_label.yml
ddev drush cache:rebuild

After copying the file, make the following changes:

  1. Remove the following keys: uuid, langcode, status, dependencies, field_plugin_method, cck_plugin_method, and migration_group.
  2. Add two migration tags: base_field_override and tag1_configuration.
  3. Add key: migrate under the source section.

Now, execute the migration:

ddev drush migrate:import upgrade_d7_node_title_label

You should see the following notice in the console: Processed 8 items (1 created, 0 updated, 3 failed, 4 ignored). There will also be this error: upgrade_d7_node_title_label migration: 3 failed. The eight items correspond to the eight content types from Drupal 7. For each of them, the migration will fetch the Drupal 7 content type label and create a base_field_override configuration if it differs from the default Title value. The 1 created corresponds to the Venue content type. If you go to https://migration-drupal10.ddev.site/admin/structure/types/manage/venue, the Title field label shall now be set to Name just like in Drupal 7. The 3 failed correspond to the three content types that were skipped in the upgrade_d7_node_type. If the content type does not exist in Drupal 10, it is not possible to override the title label. The 4 ignored corresponds to other four content types that were migrated: Article, Basic page, Event, and Session. Because they use Title as the Title field label, there is no need to provide a base_field_override configuration for them.

If we execute the ddev drush migrate:messages upgrade_d7_node_title_label command, we will see there are three messages for this migration. The level 1 for Source ID(s) speaker, sponsor, and swag indicate an error as described in the Message column.

Console notice

What about the four records that were ignored? They are explicitly skipped in the migration:

process:
  label:
    -
      plugin: static_map
      source: title_label
      bypass: true
      map:
        Title: 0
    -
      plugin: skip_on_empty
      method: row

The process pipeline for the label destination property skips the whole row if the source title_label has a value of Title. It first uses the static_map plugin to convert the label Title to the number zero. This is later passed to the skip_on_empty plugin, which treats the zero as an empty value. The whole row is skipped by method: row.

Ideally, everytime we use skip_on_empty or skip_on_value to skip a whole row, we should set the message configuration option. That way, the map table for this migration stores the reason why a row was skipped. See this issue on improving exception messages in process plugins.

In the meantime, because we are customizing the migrations, we could add the message configuration option to the skip_on_empty plugin ourselves. Doing so will be left as an independent exercise for you to do. To give it a try, rollback the migration, add the message configuration, import the migration again, and review the messages.

Technical note: The migrate messages table is truncated when the migration is rolled back. If you execute migrate:messages right after migrate:rollback there will not be any messages. You need to run migrate:import again for messages to appear, when properly configured.

Striking a balance

We added a whole new migration to update the Title field label of the Venue content type. Was that really needed? Probably not. We could have made this change directly in Drupal 10 after the Venue content type was migrated. We did it here to illustrate a point.

Multiple migrations can affect the same concept in Drupal 10. In this case, both upgrade_d7_node_type and upgrade_d7_node_title_label affect how content types are imported. Under the hood, they create different configuration entities, but the point still stands. Another example of this is migration fields which consist of four different migrations: storage, instance, widget, and formatter. We are going to cover field migrations in future articles.

That being said, we need to strike a balance between what to automate and what changes to do manually in Drupal 10. In cases like the above, it will be evident that updating the configuration manually is better. In other cases, like migrating fields, it is beneficial to run an automated migration even if the results are not 100% perfect. Past experience and familiarity with writing custom migrations will go a long way in making the right decision.

In my experience, menus and blocks are rarely migrated automatically. For permissions, an initial automated migration is performed and they are later adjusted manually. For fields, try to migrate as much as possible automatically. In a field migration for a recent project I was able to automate: dropping fields, renaming fields, moving fields from one entity to another, changing field types, changing field widgets/formatters, and more. This required a good amount of customizations, but it was worth doing considering the number of fields to migrate and the content model changes that needed to be applied. Explaining how this was accomplished is outside the scope of this series, but we will touch on some techniques that could be used to perform these and other customizations.

Ultimately, our goal is to teach you how to create custom migrations for your own projects.

What about configuration changes and version control?

We are not going too deep into configuration management nor version control in this series. Those are very important topics to carefully consider in your real life projects. In the interest of brevity, we focus on explaining how to write custom migrations.

That being said, below are some tips for you to consider:

  • When executing migrations that create configuration, export that configuration and commit it to the repository.
  • When a configuration migration is rolled back and imported again, the uuid key of the *.yml files will change. You might want to wait for all/most customizations to be completed before committing the resulting configuration.
  • If you reinstall the site from the existing configuration, the _core/default_config_hash value will change. Also, the migration map and messages tables will be lost. For configuration migrations, this is acceptable as long as your content migrations do not perform lookups against configuration migrations. We are going to follow this approach during the series.
  • A single migration can produce multiple configuration entities. In today's example, the upgrade_d7_node_type migration created 5 node.type.*.yml configuration entities.
  • When customizing migrations, it is a good idea to commit the generated migration as-is. Then, add the modifications to the migration files as separate commits. This will make it easier to identify what changes were made.

In the next article, we will learn to migrate taxonomy vocabularies and convert Drupal 7 field collections to Drupal 10 paragraphs.

Contact us to learn how we can help you with your migration.

Schedule a Free Consultation

Image by Malcolm Brook from Pixabay