Series Overview & ToC | Previous Article | Next Article

Article Content:

In the previous article, we learn about Drupal fields, their underlying data structure at the database level, and some considerations for creating custom field migrations. We also pointed out that migrating fields require at least four different migrations. Today, we will start the multi-step process of migrating fields by taking care of the first two components: storage and instance settings.

We are going to initially run the automated migrations with no significant modifications to familiarize ourselves with the process and to identify some of the errors that come up when migrating fields. Then, we are going to use the Migrate Skip Fields module and perform manual configuration updates to our field configuration to account for content model changes.

Before we begin

Our example project involves many content model changes that require adjustments to field configurations. Refer to article 8 for a high level overview of the migration plan. Also refer to the field_instance tab of the Drupal Site Audit Template for a summary of the content model changes related to fields.

In article 14, we talked about exporting and committing configuration changes to version control. This is useful in case something goes wrong and you need to revert to a previous state of the site configuration. You could also go one step further and create a full database dump after completing or before starting a migration. Importing a database dump is generally easier than restoring a previous state of the configuration, especially if you have uncommitted configuration files. It also has the added benefit of including any content that exists on the site. Some content migrations can take a very long time to complete. Having a snapshot of the (partially) migrated database makes it easier to start from a known checkpoint in case something goes wrong with the next migration. You can use the following DDEV commands to create, list, restore, and delete database backups.

# Create database backup.
ddev snapshot --name [BACKUP_NAME]

# List database backups.
ddev snapshot --list

# Restore database backup.
ddev snapshot restore [BACKUP_NAME]

# Delete all database backups.
ddev snapshot --cleanup

# Delete single database backup.
ls .ddev/db_snapshots/
rm [FILENAME]

Replace [BACKUP_NAME] by an identifier using underscores (_) instead of spaces ( ) to separate words. The commands above were provided because the example project is set up with DDEV already. If preferred, you could use other tools to create and restore database dumps. For example, drush sql:dump to create a backup and ddev import-db to restore it.

Technical note: The snapshot command in DDEV backs up the entire state of the database server for the project. If you have two or more databases, all of them will be included in the backup. When restoring, all databases will be recreated based on the content of the backup file. Snapshots are stored in the .ddev/db_snapshots folder as gzipped files. They can be manually and individually deleted if needed.

Field storage and instance configuration

Drupal fields consist of four different configuration elements: storage, instance, widget, and formatter settings. We are migrating the first two today. As a quick reminder, field storage settings are closely related to the underlying structure Drupal uses to save information at the database level. If you re-use fields across multiple bundles of the same entity, all those fields will share the same storage configuration. On the other hand, field instance settings can vary per entity/bundle combination. In many cases, field instance settings are related to validation logic when collecting data. Refer to the previous article for a more thorough explanation of Drupal fields.

We use upgrade_d7_field and upgrade_d7_field_instance to migrate field storage and instance settings respectively. Let's copy these migrations from the reference folder into our custom module and rebuild caches for the migrations to be detected.

cd drupal10
cp ref_migrations/migrate_plus.migration.upgrade_d7_field.yml web/modules/custom/tag1_migration/tag1_migration_config/migrations/upgrade_d7_field.yml
cp ref_migrations/migrate_plus.migration.upgrade_d7_field_instance.yml web/modules/custom/tag1_migration/tag1_migration_config/migrations/upgrade_d7_field_instance.yml
ddev drush cache:rebuild

Note that while copying the files, we also changed their names and placed them in a migrations folder inside our custom module. After copying the files, make the following changes:

  • Remove the following keys: uuid, langcode, status, dependencies, cck_plugin_method, and migration_group. Notice that for field migrations, we preserve the field_plugin_method key.
  • Add the field_storage_config tag to the upgrade_d7_field migration and the field_config to the upgrade_d7_field_instance. Add the tag1_configuration tag to both migrations.
  • Add key: migrate under the source section.
  • Update the required migration_dependencies to include the migrations that create entities. Given that fields are attached to entities, we want to be explicit about establishing dependencies among migrations. In this case, add upgrade_d7_node_type, upgrade_d7_taxonomy_vocabulary, and upgrade_d7_field_collection_type to both field storage and instance migrations.

After the modifications, the upgrade_d7_field.yml file should look like this:

id: upgrade_d7_field
class: Drupal\migrate_drupal\Plugin\migrate\FieldMigration
field_plugin_method: alterFieldMigration
migration_tags:
 - 'Drupal 7'
 - Configuration
 - field_storage_config
 - tag1_configuration
label: 'Field configuration'
source:
 key: migrate
 plugin: d7_field
 constants:
   status: true
   langcode: und
process:
 entity_type:
   -
     plugin: get
     source: entity_type
   -
     plugin: static_map
     map:
       field_collection_item: paragraph
       paragraphs_item: paragraph
     bypass: true
 status:
   -
     plugin: get
     source: constants/status
 langcode:
   -
     plugin: get
     source: constants/langcode
 field_name:
   -
     plugin: get
     source: field_name
 type:
   -
     plugin: process_field
     source: type
     method: getFieldType
     map:
       d7_text:
         d7_text: d7_text
       taxonomy_term_reference:
         taxonomy_term_reference: taxonomy_term_reference
       image:
         image: image
       email:
         email: email
       field_url:
         field_url: field_url
       addressfield:
         addressfield: addressfield
       telephone:
         telephone: telephone
       entityreference:
         entityreference: entityreference
       file:
         file: file
       list:
         list: list
       datetime:
         datetime: datetime
       number_default:
         number_default: number_default
 cardinality:
   -
     plugin: get
     source: cardinality
 settings:
   0:
     plugin: d7_field_settings
   field_collection:
     plugin: field_collection_field_settings
destination:
 plugin: 'entity:field_storage_config'
migration_dependencies:
 required:
   - upgrade_d7_field_collection_type
   - upgrade_d7_node_type
   - upgrade_d7_taxonomy_vocabulary
 optional: {  }

And the upgrade_d7_field_instance.yml file should look like this:

id: upgrade_d7_field_instance
class: Drupal\migrate_drupal\Plugin\migrate\FieldMigration
field_plugin_method: alterFieldInstanceMigration
migration_tags:
 - 'Drupal 7'
 - Configuration
 - field_config
 - tag1_configuration
label: 'Field instance configuration'
source:
 key: migrate
 plugin: d7_field_instance
 constants:
   status: true
   comment_node: comment_node_
process:
 type:
   -
     plugin: process_field
     source: type
     method: getFieldType
 entity_type:
   -
     plugin: get
     source: entity_type
   -
     plugin: static_map
     map:
       field_collection_item: paragraph
       paragraphs_item: paragraph
     bypass: true
 field_name:
   -
     plugin: get
     source: field_name
 bundle_mapped:
   -
     plugin: static_map
     source: bundle
     bypass: true
     map:
       comment_node_forum: comment_forum
   -
     plugin: paragraphs_process_on_value
     source_value: entity_type
     expected_value: field_collection_item
     process:
       plugin: paragraphs_strip_field_prefix
 _comment_type:
   -
     plugin: explode
     source: bundle
     delimiter: comment_node_
   -
     plugin: extract
     index:
       - 1
     default: false
   -
     plugin: skip_on_empty
     method: process
   -
     plugin: migration_lookup
     migration: {  }
 bundle:
   -
     plugin: field_bundle
     source:
       - entity_type
       - '@bundle_mapped'
 label:
   -
     plugin: get
     source: label
 description:
   -
     plugin: get
     source: description
 required:
   -
     plugin: get
     source: required
 status:
   -
     plugin: get
     source: constants/status
 allowed_values:
   -
     plugin: sub_process
     source: allowed_vid
     process:
       -
         plugin: migration_lookup
         migration: upgrade_d7_taxonomy_vocabulary
         source: vid
 settings:
   0:
     plugin: d7_field_instance_settings
     source:
       - settings
       - widget
       - field_definition
   field_collection:
     plugin: field_collection_field_instance_settings
 default_value_function:
   -
     plugin: get
     source: ''
 default_value:
   -
     plugin: d7_field_instance_defaults
     source:
       - default_value
       - widget
 translatable:
   -
     plugin: get
     source: translatable
destination:
 plugin: 'entity:field_config'
migration_dependencies:
 required:
   - upgrade_d7_field
   - upgrade_d7_field_collection_type
   - upgrade_d7_node_type
   - upgrade_d7_taxonomy_vocabulary
 optional: {  }

Now, rebuild caches for our changes to be detected and execute the migrations. RWe run migrate:status to make sure we can connect to Drupal 7. Then, we run migrate:import to perform the import operations. Notice that both commands allow specifying multiple migrations when separated by a comma without spaces between migration IDs. When you specify multiple migrations to be executed, the command will build a dependency tree and execute the migrations trying to respect what is specified in the migration_dependencies key.

ddev drush cache:rebuild
ddev drush migrate:status upgrade_d7_field,upgrade_d7_field_instance
ddev drush migrate:import upgrade_d7_field,upgrade_d7_field_instance

After executing the migrate:import operation, we encounter our first error:

[error]  Attempt to create a field storage field_video_recording with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:271)
[notice] Processed 28 items (27 created, 0 updated, 1 failed, 0 ignored) in 0.4 seconds (4578.2/min) - done with 'upgrade_d7_field'
In MigrateRunnerCommands.php line 414:
 upgrade_d7_field migration: 1 failed.

While it may sound obvious, it is very important to carefully read all messages logged to the console, especially when there are errors. Notice that the error states a problem with the upgrade_d7_field migration, but there is no reference to upgrade_d7_field_instance. If one of the migrations triggers an error, the drush migrate:import command exits immediately and any subsequent migrations are not executed.

For now, ignore the error and try to import field instances again by running ddev drush migrate:import upgrade_d7_field_instance. You will get a different set of errors:

[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id speaker. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Attempted to create, modify or delete an instance of field with name field_video_recording on entity type node when the field storage does not exist. (/var/www/html/web/core/modules/field/src/Entity/FieldConfig.php:316)
[error]  Missing bundle entity, entity type node_type, entity id sponsor. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id sponsor. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id swag. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id swag. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[error]  Missing bundle entity, entity type node_type, entity id swag. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityType.php:916)
[notice] Processed 34 items (20 created, 0 updated, 14 failed, 0 ignored) in 0.3 seconds (6051.3/min) - done with 'upgrade_d7_field_instance'
In MigrateRunnerCommands.php line 414:
 upgrade_d7_field_instance migration: 14 failed.

Let's analyze the output of executing both migrations. In the case of upgrade_d7_field, we get a very useful error message: Attempt to create a field storage field_video_recording with no type. This means that the field type of the field_video_recording is not supported in Drupal 10. Technically speaking, in Drupal 10 there is no field plugin with the same machine name as the field type name in Drupal 7 nor a field migrate plugin that maps the field type from the source site to one available in the target site. This migration also produced the following notice: Processed 28 items (27 created, 0 updated, 1 failed, 0 ignored). 27 out of 28 fields created without errors is not bad at all. It is important to note that this field storage settings was created for all entities that had field attachments in Drupal 7. In our case, that includes content types and paragraph types.

As for the upgrade_d7_field_instance migration, the error messages are a bit more cryptic. The error Missing bundle entity, entity type node_type, entity id speaker. means that we are trying to migrate a field that belongs to an entity/bundle combination that does not exist in Drupal 10. In our example, this is the result of skipping some content types in the upgrade_d7_node_type migration. Any Drupal 7 field attached to a content type that was skipped in Drupal 10 will yield this error. The other error — Attempted to create, modify or delete an instance of field with name field_video_recording on entity type node when the field storage does not exist. — is more self explanatory. A field instance setting depends on the corresponding field storage setting to exist. In the upgrade_d7_field, it was not possible to migrate the storage settings for the field_video_recording; therefore, it is not possible to migrate the instance settings for a non-existent field. The field instance migration produced the following notice: Processed 34 items (20 created, 0 updated, 14 failed, 0 ignored). A higher percentage of failed records, but that is expected given we skipped three content types will all their fields as part of our custom migration. All the fields attached to those content types will yield an error when trying to migrate them.

Making sense of error messages might be challenging. Reading online documentation, asking for support in community channels, following guides like this series, debugging migrations, and a lot of practice will help you understand the different error messages that you will inevitably encounter in real life migration projects.

To get more context on a failed record, check the migration messages logged during the import operation. You can do that using the drush migrate:messages command or the migration messages admin interface. Executing the ddev drush migrate:messages upgrade_d7_field command we get... nothing. Even though there were records that failed to process, the migrate API did not log the reasons they failed. The only information available is the error message printed to the console. . To find the source of the error, we can search the codebase for the error message. In this case, the error is produced in the \Drupal\field\Entity\FieldStorage class. While this technical detail might not provide a lot of context, it is always helpful to identify where errors originate. This helps in understanding why it happens and makes it easier to find a solution.

Now we will check if the upgrade_d7_field_instance migration logged any errors. Feel free to execute the ddev drush migrate:messages upgrade_d7_field_instance command and see the results in the command line. Alternatively, go to https://migration-drupal10.ddev.site/admin/reports/migration-messages and click on the link indicating the message count for the upgrade_d7_field_instance.

Message Count

The report provides more context as to why an error was thrown. Namely, you can see which entity type / bundle / field name combination triggered the error. Here, all the Missing bundle entity, entity type node_type, entity id errors are related to fields attached to content types we explicitly skipped: speaker, session, and swag. We will learn how to prevent these errors from happening, but for now they are to be expected.

Before moving forward, let's review the result of our migrations. The migrations reported that most field storage and instance settings were created. You can go to the content types and paragraph types administration pages and review the fields attached to them. For example, below is a screenshot of the fields in the Event content type.

Event Content Type

The good news is that many fields are already added to the content type. The bad news is that we received a couple of cryptic error messages. Again, prior experience and some intuition will go a long way into understanding and fixing these errors. Pay close attention to the Field type configuration for field_sponsors. It says it is an entity reference field, allowing references to nodes (content), of type... nothing? If we click on the edit button, we see that there is no content type selected. When we review this field in Drupal 7, we notice that it allows referencing nodes of type Sponsor, but we skipped that content type altogether in Drupal 10. Sponsor nodes are going to be migrated as taxonomy vocabularies. Therefore, we can update this field configuration to point to taxonomy terms instead of nodes. In the Field Storage section, change Type of item to reference to Taxonomy term. In the Reference type section select the Sponsor vocabulary. Save the field configuration and the errors we saw before will be gone.

If you want to practice, fix a similar error in the Session content type. These errors were the result of a mismatch between the content model in Drupal 7 compared to the content model in Drupal 10. After executing each migration, it is good practice to verify it is producing the expected results. In the case of the upgrade_d7_field and upgrade_d7_field_instance migrations, this means reviewing the field attached to all entity/bundle combinations.

Before continuing, let me share one of my favorite administration pages in all of Drupal: the field list report. Go to https://migration-drupal10.ddev.site/admin/reports/fields and you will be greeted by a lot of errors. At this point, some of these errors should start to feel familiar. Let's ignore the errors for now and have a look at the report itself.

Report

You will see every field attached to every entity type. For each field, there is the field type and the module that provides it. The Used in column indicates all the bundles of an entity type where a field is used. The Summary column will include extra information about the field. For example, for entity reference fields, you will see which entity type the field allows to reference. The ideal scenario is that you do not see any blank spaces in the Used in column. If there is a row without a value in Used in or if there are commas without text in between like Session, Event, , this is a signal that something went wrong with the field instance migrations.

Rolling back migrations

Migrations are an iterative process. It is common to update a migration multiple times until you get the desired results. That process requires rolling back and importing the migration again. Normally, rolling back a migration does not affect content or configuration that was imported by a different migration. But that is not the case with field-related migrations. If we were to rollback the field instance migration, the operation also deletes some of the field storage settings.

Consider the following with regards to rolling back migrations:

  • The migration destination plugin determines the action to take when a migration is rolled back. For example, when using the migrating nodes with revisions and translations, the migrate API would not delete a node if there are still revisions or translations in the system. This is done on a per row basis, not for the migration as a whole.
  • When rolling back a migration, other Drupal APIs are invoked. In the case of content and configuration migrations, Drupal would call the entity delete operations, where many checks and actions are performed before and after an entity is deleted. For example, the field instance post-delete actions deletes the field storage configuration if the field is not used by other other bundles and it is not marked as persistent.
  • Some configuration attached to an entity might prevent it from being deleted. For example, a field storage setting can be marked as persistent meaning it would not be deleted even if it is no longer used. An example of this is the node body field storage, which gets reused when a new content type is added via the user interface.

Getting back to field-related migrations, if you rollback the upgrade_d7_field_instance migration, we know that at least some field storage settings will also be removed. But, if you were to run ddev drush migrate:status upgrade_d7_field you would see that it reports that all fields had been migrated and that there are no records to process. This means that field storage settings were deleted from the system, but the Migrate API is not aware of that. This happens because the migration system keeps its own mapping database tables to keep track of the status of each migration. In this case, rolling back the upgrade_d7_field_instance migration did not affect the mapping tables of the upgrade_d7_field migration. So, to work around this, we need to rollback both upgrade_d7_field_instance and upgrade_d7_field, and import them again. Otherwise, when trying to import field instance settings you will get errors related to missing field storage settings. Trying this out will be left as an exercise to the reader.

Skipping fields in Drupal migrations

The content model changes include dropping a whole content type, changing from one entity to another, and changing from one field type to another. This means that a one-to-one copy of the Drupal 7 field configuration is not possible. To make the transition easier, we are going to prevent some fields from being migrated. Then, we will manually create or update field settings as needed to match the new content model. The Migrate Skip Fields module can be used to prevent migrating fields that are not necessary. Make sure to include the module via Composer and enable it.

ddev composer require 'drupal/migrate_skip_fields:^1.0@alpha'
ddev drush pm:enable migrate_skip_fields

Once enabled, you can skip fields by entity, bundle, name, or type with the following settings:

// Drupal core version of source site. Either '6' or '7'.
$settings['migrate_skip_field_source_version'] = '7';
// Skip by entity type, bundle, and field name. An asterisk serves as a wildcard to skip all values of one component.
$settings['migrate_skip_fields_by_name'] = [
 'entity_type:bundle:field_name',
 'entity_type:bundle:*',
 'entity_type:*:field_name',
 '*:*:field_name',
];
// Skip by field type. See hook_field_info. Alternatively, see the content_node_field table in Drupal 6 or the field_config table in Drupal 7.
$settings['migrate_skip_fields_by_type'] = [
 'field_type_1',
 'field_type_2',
];

The migrate_skip_fields_by_name setting allows specifying multiple values each using three components (entity type, bundle, and name) separated by a colon (:) to indicate which fields should be skipped. You can use asterisk (*) as a wildcard for any of the components. That means that you want to skip fields for all values of that component. For example, node:swag:* means that you want to skip all fields attached to the swag bundle (content type) of the node entity type. For migrate_skip_fields_by_type, you need to use the machine name of the Drupal 7 field type to exclude. This can be found in the hook_field_info implementation of the module that provides the field. Alternatively, you can query the field_config table in the Drupal 7 database to find details on field machine names and their types. For those following the series, article 8 included a Drupal 7 site audit in which we populated a source site template introduced in article 3. You can use the field_instance tab of that document as a reference of what we need to do in our example project.

The above settings need to be added in settings.php or in a file included from it. In our example project, we use settings.migrate.php for migration-related settings. Add the following settings to that file:

// Migrate skip fields settings.
$settings['migrate_skip_fields_source_version'] = '7';
$settings['migrate_skip_fields_by_name'] = [
 'node:speaker:*', // Changed to user entity.
 'node:sponsor:*', // Changed to taxonomy term entity.
 'node:swag:*', // Content type no longer needed.
 '*:*:field_image', // Changed to media reference field.
];
$settings['migrate_skip_fields_by_type'] = [
 'youtube', // Changed to media reference field.
];

The above translates to skipping all fields attached to the speaker, sponsor, and swag content types; all fields whose machine name is field_image no matter the entity type or bundle; and all fields of type youtube. Later in the series, we will use a combination of Drupal recipes and manual updates to bring back some of the fields that were skipped with the updated configuration for Drupal 10.

The Migrate Skip Fields module acts on the import operation of the field-related migrations. Therefore, we need to rollback and import them again.

ddev drush migrate:rollback upgrade_d7_field,upgrade_d7_field_instance
ddev drush cache:rebuild
ddev drush migrate:import upgrade_d7_field,upgrade_d7_field_instance

There should not be any failed records in the import operation report. Use the drush migrate:messages command or the migration messages admin interface to see helpful log data added by the Migrate Skip Fields module. Then, visit the field list report at https://migration-drupal10.ddev.site/admin/reports/fields Notice that the number of fields is significantly lower and that there are no error messages at the top of the page. A big win already, but we are not done yet.

Remember manual configuration updates we did to the field_sponsors field in the Event content type earlier? Because we rolled back the migration and the field was deleted, we need to do these manual updates again. As mentioned before, after each field migration, you should review the associated entities and bundles to verify the migrated configuration is free of errors and in line with the new content model.

Let's make the following updates:

  • In the Event content type, edit the field_sponsors field. Change Type of item to reference to Taxonomy Term and for Vocabulary select Sponsor. Then save the field configuration.
  • In the Session content type, edit the field_speakers field. Change Type of item to reference to User and for Sort by select Name. Then save the field configuration.

With these two manual updates complete, we have finished migrating field storage and instance settings. What a journey! In your own projects, be sure to export the new, fleshed out configuration and commit it to version control. In our next articles, we will walk through migrating field widgets, formatters, and groups.

Image by Valdas Miskinis from Pixabay