Importing large data sets into ServiceNow and transforming them can be a time-consuming process. Starting with the Madrid release, ServiceNow introduced the Concurrent Import Set feature, which presents a way to essentially “multi-thread” data imports. This functionality is available as a baseline feature and is fairly simple to leverage.
Setup:
There is one element of configuration that needs to be addressed prior to kicking off a concurrent import set. You must manually configure the distribution of Import Set Transformer records across ServiceNow nodes.
- Navigate to the “sys_trigger” table (list view)
- filter by: Name = “Import Set Transformer“
- Add the “System ID” field to the list
- Set the “System ID” on one of the records to “Active Node“
- Set the “System ID” on the other available records by splitting up the available nodes
- example 1:If there are 10 Import set Transformer records left and you have (2) active nodes, set (5) of the records to Node 1 and (5) of the records to Node 2)
- example 2:If there are 10 import set Transformer records left and you have (1) active node, set (10) records to that node – See screenshot
Calling a Concurrent Import
The only way to make an import run in concurrent threads is to schedule it as a scheduled import.
- Navigate to Scheduled Import Sets > Administration > Scheduled Imports
- Create New
- Provide Name
- Select the Data Source you want to import and transform
- Schedule the Runtime/Frequency
- Check the “Concurrent Import” checkbox.
- The default partition method is round robin (this will alternate records being imported between each of the threads you just created)
- You can also create a custom script to feed records in a different order if you’d like.
Your Scheduled import should look something like this:
Running a Concurrent Import set:
- To test this, you can use the “Execute Now” UI Action (or wait until the scheduled import kicks off).
- This will create a (parent) “concurrent import set” along with multiple (dependent on the number of threads you set up) child Import Sets
- With the round robin setting, the total number of records on the import, will be split evenly among the available threads (each having its own Import Set) , so that all threads finish within a relatively similar time-frame
- Once all of the imports threads are completed, the multi threaded transforms will start.
- You can view the status of the individual imports and transform histories by clicking on the System Import Sets > Advanced > Concurrent Import Sets (module) in the left navigation menu. And the results will look something like this.
