Configuring a concurrent import set in ServiceNow

Importing large data sets into ServiceNow and transforming them can be a time-consuming process. Starting with the Madrid release, ServiceNow introduced the Concurrent Import Set feature, which presents a way to essentially “multi-thread” data imports. This functionality is available as a baseline feature and is fairly simple to leverage.

Setup: 

There is one element of configuration that needs to be addressed prior to kicking off a concurrent import set. You must manually configure the distribution of Import Set Transformer records across ServiceNow nodes.

• Navigate to the “sys_trigger” table (list view)
  • filter by: Name = “Import Set Transformer“
• Add the “System ID” field to the list

• Set the “System ID” on one of the records to “Active Node“

• Set the “System ID” on the other available records by splitting up the available nodes
  • example 1:If there are 10 Import set Transformer records left and you have (2) active nodes, set (5) of the records to Node 1 and (5) of the records to Node 2)
  • example 2:If there are 10 import set Transformer records left and you have (1) active node, set (10) records to that node – See screenshot

Calling a Concurrent Import

The only way to make an import run in concurrent threads is to schedule it as a scheduled import.

• Navigate to Scheduled Import Sets > Administration > Scheduled Imports
• Create New
  • Provide Name
  • Select the Data Source you want to import and transform
  • Schedule the Runtime/Frequency
  • Check the “Concurrent Import” checkbox.
    • The default partition method is round robin (this will alternate records being imported between each of the threads you just created)
    • You can also create a custom script to feed records in a different order if you’d like.

Your Scheduled import should look something like this:

Running a Concurrent Import set: 

• To test this, you can use the “Execute Now” UI Action (or wait until the scheduled import kicks off).
• This will create a (parent) “concurrent import set” along with multiple (dependent on the number of threads you set up) child Import Sets
• With the round robin setting, the total number of records on the import, will be split evenly among the available threads (each having its own Import Set) , so that all threads finish within a relatively similar time-frame
• Once all of the imports threads are completed, the multi threaded transforms will start.
• You can view the status of the individual imports and transform histories by clicking on the System Import Sets > Advanced > Concurrent Import Sets (module) in the left navigation menu. And the results will look something like this.