Handle large datasets

<< Click to Display Table of Contents >>

Navigation:  How do I? >

Handle large datasets

Large datasets (e.g. with millions of rows) can slow down processing. If slow processing is a problem you can:

Add a sample transform straight after the input and set Rows to pass through only the first 100 or so rows. Once you have completed all your transforms you can then Disable sampling to pass through all rows.

Set Maximum memory usage to a higher value in the Preferences window, to ensure you don't run out of memory.

Set Optimize processing for to Minimum memory use in the Preferences window, to reduce memory usage.

Uncheck Run>Auto Run to give yourself full control over processing.

Auto Run unchecked

or

Check Run>Auto Run and:

oSet Right pane processing delay in the Preferences window to a longer time (say 5 seconds) to ensure that changes aren't processed until you have finished making the changes.

oSet Write mode to Disabled in output files, until you are ready to write them.

 

You can check View>Timing Profile to see where the processing time is being spent.

 

Timing profile

 

See also:

Handle datasets with many columns

Manage a large number of Center pane items

Processing