Infer Schema Type Transform

Infer Schema Type examines column values and converts each selected column to a more suitable data type. The result can represent the data more accurately and use less memory.

Basic Usage

To infer column types:

1. Select the Infer Schema Type transform from the transform menu.
2. Choose one or more fields under Columns to infer. Leave it empty to infer every column.
3. Apply the transformation.

note

The panel shows the names of the columns that will be processed.

Prepare categorical columns for Encode

If Encode reports that no categorical columns were found, connect Infer Schema Type upstream and apply it first. Reopen Encode, then select the newly inferred categorical columns. See the Encode Transform.

Configuration Options

Basic Options

• Columns to infer: Select the columns to inspect. Their names appear in the panel.

tip

Leave Columns to infer empty to inspect every column.

How It Works

For each included column, the transform can:

1. Detect dates and times: Convert parseable string columns to datetime values.
2. Convert numbers: Convert numeric strings to an integer or floating-point type.
3. Identify categories: Convert columns that appear to contain categorical data.
4. Downcast numbers: Use a smaller numerical type when possible to save memory.

Examples

Example: Inferring Types in a Sales Dataset

Input Dataset:

date_column	season_column	sales_column
October 10, 2023	Fall	123.45
10/31/2023	Fall	234.56
November 15, 2023	Fall	345.67
12/31/2023	Winter	678.90

Initial Data Types:

• date_column: object (string)
• season_column: object (string)
• sales_column: object (string)

Configuration:

• Columns to Consider: All columns

Result:

date_column	season_column	sales_column
2023-10-10	Fall	123.45
2023-10-31	Fall	234.56
2023-11-15	Fall	345.67
2023-12-31	Winter	678.90

Inferred Data Types:

• date_column: datetime64[ns]
• season_column: category
• sales_column: float32

The transform converts the dates to datetime values, the seasons to categories, and the sales values to a numerical type.

Best Practices

1. Review every inferred type and compare it with what the column represents.
2. Override the result when your knowledge of the data calls for a different type.
3. Clean mixed-type columns first if they are not inferred as expected.
4. Keep a copy of critical source data before changing types.
5. Check that numerical downcasting has not removed precision needed for analysis.

Troubleshooting

• A date/time column is not recognized: Check for inconsistent date formats.
• A column is not converted as expected: Look for outliers, mixed types, or inconsistent values that may block inference.
• A high-cardinality column is not identified as categorical: Set its type to Categorical manually.