Using the new Drupal 8 Migration API / Module · Blog

We at Liip AG believe, that the migration API is the best and most efficient way to import data into Drupal. Here are some reasons, why you should use migrate instead of the feeds module or any other custom importer modules:

Since Drupal 8, Migrate API is part of Drupal core
Migrate will be maintained and supported as long as Drupal 8 exists as it provides the upgrade path for older Drupal versions to Drupal 8
Migrate is sponsored by Acquia and mainly supported by Mike Ryan, a well-known and skilled Drupal developer.
Migrate has out of the box support for all important Drupal objects such as nodes, users, taxonomy terms, users, files, entities and comments.
Migrate has a Drush integration, that allows you, to run import tasks from command-line or via cron job
Migrate maintains a mapping-table, has rollback functionality and even supporting a highwater field, that allows to import only new or changed datasets.
Migrate is well documented and there is an example module.

Getting started with Drupal 8 Migrate Module

The Migrate 8 module in core is only an API. There is no user interface. This makes it difficult for new developer to start with Migrate 8.

I suggest you, to install the below listed extension modules right away before you start developing if you want to realize the full potential of migrate:

Migrate Plus ( drupal.org/project/migrate_plus)

Extends the migration framework with groups
Delivers a well documented example module

Migrate Tools ( drupal.org/project/migrate_tools)

Provides Drush commands for running and managing migrations in Drupal 8

Migration Source Plugins

Importing CSV Files ( drupal.org/project/migrate_source_csv)
Importing XML ( drupal.org/project/migrate_source_xml)
Importin JSON ( drupal.org/project/migrate_source_json)

Installing a fully working Drupal 8 Migrate setup using composer

Instead of starting now to download in install all the module mentioned above, you can use my installation profile based on a composer.json file. Because a lot of modules with specific version are involved, I have prepared a fully working migrate example environment for a Liip hackday.

If you want to quickly start with the migrate module, head over to my github repository and install Drupal 8 Migrate using composer and drush . You just need to follow the instruction in the README.md

github.com/ayalon/drupal8-migrate

Comparing Drupal Migrate 7 with Drupal Migrate 8

Some of you might already have used Migrate 7. A traditional mapping was done in the constructor of a Migration class:

public function __construct($arguments) {
  parent::__construct($arguments);
  $this->description = t('Page Placeholder import');

  // Set up our destination - nodes of type migrate_example_beer.
  $this->destination = new MigrateDestinationNode('page');

  $this->csvFile = DRUPAL_ROOT . '/docs/navigation.csv';

  $this->map = new MigrateSQLMap($this->machineName,
    array(
      'ID' => array(
        'type' => 'int',
        'unsigned' => TRUE,
        'not null' => TRUE,
      ),
    ),
    MigrateDestinationNode::getKeySchema()
  );

  $this->source = new MigrateSourceCSV($this->csvFile, array(), array('header_rows' => 1, 'delimiter' => ';'));

  // Force update.
  $this->highwaterField = array();

  // Mapped fields.
  $this->addFieldMapping('title', 'name');
  $this->addFieldMapping('uid')->defaultValue(1);
  $this->addFieldMapping('status')->defaultValue(1);
  $this->addFieldMapping('promote')->defaultValue(0);
  $this->addFieldMapping('sticky')->defaultValue(0);
  $this->addFieldMapping('language')->defaultValue('de');

  // Unmapped destination fields.
  $this->addUnmigratedDestinations(array(
    'body:format',
    'changed',
    'comment',
    'created',
    'is_new',
    'log',
    'revision',
    'revision_uid',
    'tnid',
    'translate',
  ));

}

In Migrate 8 this format has been replaced with yaml files. The same mapping as above looks like that in Drupal 8:

# Migration configuration
id: page_node
label: Dummy pages
migration_group: liip
source:
 plugin: page_node
 # Full path to the file. Is overridden in my plugin
 path: public://csv/navigation_small.csv
 # The number of rows at the beginning which are not data.
 header_row_count: 1
 # These are the field names from the source file representing the key
 # uniquely identifying each node - they will be stored in the migration
 # map table as columns sourceid1, sourceid2, and sourceid3.
 keys:
   - ID
destination:
 plugin: entity:node
process:
 type:
   plugin: default_value
   default_value: page
 title: name
 uid:
   plugin: default_value
   default_value: 1
 sticky:
   plugin: default_value
   default_value: 0
 status:
   plugin: default_value
   default_value: 1
 promote:
   plugin: default_value
   default_value: 0
 'body/value': body
 'body/summary': excerpt

#Absolutely necessary if you don't want an error
migration_dependencies: {}

Understanding the new mapping with yaml files in Migrate 8

The mapping is quite straightforward.

First you have to define your Migrate source . In the example we have used a CSV source plugin. ( drupal.org/node/2129649)
Then you can map the source fields to a Migrate destination . In our case, we use a node destination ( drupal.org/node/2174881)
You can map now all source fields to destination fields , for example you map a column of the CSV file to the node title field
Every field can be processed and modified before it is passed to the final node field. There are a lot of useful process plugins like “default_value”, “callback” or “skip_if_empty”
You can find a list of all process plugins here: drupal.org/node/2129651
Of course you can easily write your own plugins and use them while migrating data

Example: Importing a menu tree and create dummy nodes using Drupal Migrate 8

For demonstration purpose I created a small module, that reads a menu tree from a CSV file and imports into Drupal.

The module is split into 2 tasks:

Creating a page node for every row in the csv file
Create a menu item for every row and attach it to the correct node

Migrate handles these dependencies in the yaml file:

config/install/migrate_plus.migration.menu_item.yml

Dependancies

migration_dependencies:
 required:
   - page_node

See the full module code here:

github.com/ayalon/drupal8-migrate/tree/master/web/modules/custom/migrate_menu

Developing your own Drupal 8 migrate modules and fighting caching issues

You have learned, that the whole migration mapping is now done in yaml files. But how about writing your own migration yaml files?

Unfortunately, there are some pitfalls for new Drupal 8 developers. Because of the Configuration Management Interface ( drupal.org/documentation/administer/config) of Drupal 8, all yml files in the “config/install” directory are only imported when installing the module.

This is very impractical if you want to develop new configuration files. To address this, a module “Configuration Development” ( drupal.org/project/config_devel) which resolves the caching issues can be installed. It is possible to import certain yml files on every request. But unfortunately drush commands are not supported yet. So we need to add all yaml files we want to import into a new section in our module.info.yml.

config_devel:
 install:
   - migrate_plus.migration.page_node
   - migrate_plus.migration.menu_item
   - migrate_plus.migration_group.liip

Then we can run the following commands after updating the yml file. This will import the new configuration file into CMI.

drush cdi <module_name>
drush cr

In short:

You always have to remember, that you have to import the yaml files and clear the cache after changing the mapping before executing the migration again.

Testing and running your migration

If your module is set up correctly, you can run “ drush ms” to see if the migration is ready:

Now you can run the migration using

drush mi <migration_name>

If you want to update already imported items you can use the –update option:

Advanced example importing a JSON source into Drupal nodes

During the last hackday at Liip I wrote a small module that is consuming a JSON source from

jsonplaceholder.typicode.com importing some postings and comments to a Drupal 8 website.

The module is split into 3 tasks:

A feature module that installs the node type and fields called “migrate_posts”
A migration importing post nodes
A migration importing comments and attaching them to the already imported nodes

You can find the feature module here:

github.com/ayalon/drupal8-migrate/tree/master/web/modules/custom/migrate_posts

The migration module itself is here:

github.com/ayalon/drupal8-migrate/tree/master/web/modules/custom/migrate_json_hackday

Migration Process Plugin

Inside the migration module from above you will find a simple process plugin. The subject field of a comment in drupal only accepts a certain number of chars by default. Therefore I wrote a small process plugin, that truncates the incoming subject string:

subject:
 plugin: truncate
 source: name

The process plugin needs an annotation ( api.drupal.org/api/drupal/core%21core.api.php/group/annotation/8.2.x) to be picked up by the migration API. You can later refer to the id in the yaml file:

<?php

namespace Drupal\migrate_json_hackday\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\Component\Utility\Unicode;

/**
 *
 * @MigrateProcessPlugin(
 *   id = "truncate"
 * )
 */
class Truncate extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    return Unicode::truncate($value, 64);
  }
}

You can see the whole module code under:

github.com/ayalon/drupal8-migrate/tree/master/web/modules/custom/migrate_json_hackday

Final Words

At the moment, there migration is under heavy development. For Drupal 8.1 a lot of changes have been made and these changes were breaking a lot of helper modules. Especially the Migrate UI is not working since several months now under Drupal 8.1. You can see more information under drupal.org/node/2677198.

But nevertheless, Migrate 8 is getting more and more mature and stable. And it's time to learn the new yaml syntax now! If you have any suggestions, just drop a comment.

Do you have a question, a comment, or just feeling inspired? Mention us or share this article on Mastodon, Twitter or LinkedIn.