Schema drift is where the column structure of your data changes over time. For example columns are added, removed, renamed or moved to a different position in the column order. This is a frequent occurrence in the real world, where you often don’t have full control over the data format.
Schema drift can cause significant problems for automated data processing systems. However Easy Data Transform has powerful features for handling schema drift, including:
- columns changing name
- columns changing order
- columns being added
- columns being removed
You can specify the expected column names and their expected ordering in the Easy Data Transform Schema window. Just select the input item and set Schema to Yes in the right pane. You can then set-up the schema (it will default to the current schema of the input).
If the column names and ordering don’t match you can either:
- modify them to match; or
- stop processing with an error
If the schema has changed, warnings will be output to the Warnings tab of the input.
And a summary of the changes made will also be output to the Info tab of the input.
Easy Data Transform can perform schema validation on a range of formats, including Excel, CSV, JSON and XML. It can easily handle datasets with millions of rows and thousands of columns.
See the video above for more details and an example.
As well as schema validation, Easy Data Transform can help with cleaning, filtering, blending and profiling your data. All without coding. Download the free trial in a few clicks and try it yourself.
Windows Download
Mac Download