Easy Data Transform v2 improvements

This page summarises some of the improvements we have made to Easy Data Transform in v2.

---

Easy Data Transform now supports schema versioning for inputs. If the input columns differ from expected, you can automatically:

• Insert missing columns (with a default value).
• Re-order out of order columns.
• Add extra columns to the end or ignore them.

Or you can stop with an error.

---

The new Verify transform allows you to check that a dataset has expected data values.

Currently there are 48 different verification checks you can make:

• At least 1 non-empty value
• Contains
• Don’t allow listed values
• Ends with
• Integer except listed special value(s)
• Is local file
• Is local folder
• Is lower case
• Is sentence case
• Is title case
• Is upper case
• Is valid EAN13
• Is valid email
• Is valid telephone number
• Is valid UPC-A
• Match column name
• Matches regular expression
• Maximum characters
• Maximum number of columns
• Maximum number of rows
• Maximum value
• Minimum characters
• Minimum number of columns
• Minimum number of rows
• Minimum value
• No blank values
• No carriage returns
• No currency
• No digits
• No double spaces
• No duplicate column names
• No duplicate values
• No empty rows
• No empty values
• No gaps in values
• No leading or trailing whitespace
• No line feeds
• No non-ASCII
• No non-printable
• No punctuation
• No symbols
• No Tab characters
• No whitespace
• Numeric except listed special value(s)
• Only allow listed values
• Require listed values
• Starts with
• Valid date in format

You can set a Severity for each failed rule.

Verification fails are shown, color-coded, in the Right pane data table. Hover over a header or value for information on the rule(s) failed.

---

You can now easily compare the column structure in 2 or more datasets, by Column name or Column index.

---

You can now easily compare data values in 2 datasets, by Row numbers or Key columns.

An overview of the differences are shown on the left. You can click or drag this overview to navigate.

---

You can now Join 2 datasets by Row number instead of using key columns.

---

The Matching option in the Lookup transform now has additional options:

This allows you to do transforms like this:

---

The Sort transform now has additional options:

• Check intelligent sort to optionally sort dates as dates, and numbers as numbers.

• Check locale aware to use the Locale set in Preferences to determine the sorting order of text.
• Check case sensitive for case sensitive sorting of text.

---

The following operations have been added to the Calculate transform:

• DaysInMonth to calculate the number of days in the month for a date value.
• DaysInYear to calculate the number of days in the year for a date value.
• Occurrences to calculate the number of times 1 piece of text occurs in another piece of text.

---

Null characters (ASCII/Unicode value=0) are now displayed when you click show invisible:

They now are also shown separately in the Characters tab.

And you can insert Nulls into text fields, e.g. to replace them using Replace:

---

The Find Items window has additional check boxes for:

• ids
• column names
• data values

---

Added Record separator (0x1E) as an option for Line endings when outputting text files.

---

You can now filter columns by name when selecting a single column for a transform. This should make selecting columns a lot faster for datasets with lots of columns.

---

When you are outputting in Excel format, you can now check use style template to copy the styling from another Excel sheet.

Styling copied from the template sheet includes:

• column widths
• row heights
• merged cells
• cell borders
• text orientation
• fonts
• colors

---

Hovering over a connection now highlights the connection and the items at either end. This makes it easier to select connections and to trace flows.

---

Hovering over a transform button in the Left pane now highlights all transforms of that type in the Center pane. You can use this as a quick way to find transforms of a particular type.

---

View>Zoom Selected has been added. This allows you to zoom in on any selected items in the Center pane.

---

You can now hold down the Ctrl (Windows)/Cmd (Mac) key when adding a new transform or output item to add it to the bottom left of all the current items, rather than to the right of the selected items.

---

Plus lots of minor improvements.