Importing large data sets into ServiceNow and transforming them can be a time-consuming process. Starting with the Madrid release, ServiceNow introduced the Concurrent Import Set feature, which presents a way to essentially “multi-thread” data imports. This functionality is available as a baseline feature and is fairly simple to leverage.
Setup:
There is one element of configuration that needs to be addressed prior to kicking off a concurrent import set. You must manually configure the distribution of Import Set Transformer records across ServiceNow nodes.
Navigate to the “sys_trigger” table (list view)
filter by: Name = “Import Set Transformer“
Add the “System ID” field to the list
Set the “System ID” on one of the records to “Active Node“
Set the “System ID” on the other available records by splitting up the available nodes
example 1:If there are 10 Import set Transformer records left and you have (2) active nodes, set (5) of the records to Node 1 and (5) of the records to Node 2)
example 2:If there are 10 import set Transformer records left and you have (1) active node, set (10) records to that node – See screenshot
Calling a Concurrent Import
The only way to make an import run in concurrent threads is to schedule it as a scheduled import.
Navigate to Scheduled Import Sets > Administration > Scheduled Imports
Create New
Provide Name
Select the Data Source you want to import and transform
Schedule the Runtime/Frequency
Check the “Concurrent Import” checkbox.
The default partition method is round robin (this will alternate records being imported between each of the threads you just created)
You can also create a custom script to feed records in a different order if you’d like.
Your Scheduled import should look something like this:
Running a Concurrent Import set:
To test this, you can use the “Execute Now” UI Action (or wait until the scheduled import kicks off).
This will create a (parent) “concurrent import set” along with multiple (dependent on the number of threads you set up) child Import Sets
With the round robin setting, the total number of records on the import, will be split evenly among the available threads (each having its own Import Set) , so that all threads finish within a relatively similar time-frame
Once all of the imports threads are completed, the multi threaded transforms will start.
You can view the status of the individual imports and transform histories by clicking on the System Import Sets > Advanced > Concurrent Import Sets (module) in the left navigation menu. And the results will look something like this.