Types of Exceptions
In ETL workflows, exceptions are categorized into active exceptions and suppressed exceptions based on how they are managed. These types reflect the organization’s approach to handling errors, depending on whether the issues can be resolved and reprocessed or need to be deferred indefinitely.
Active Exceptions
Active exceptions are those records that fail to meet predefined rules or relationships but are flagged for immediate correction and reprocessing. These exceptions are prioritized for resolution to ensure that the affected records can be reintegrated into the ETL pipeline.
Examples of Active Exceptions:
-
Incorrect formats, such as invalid dates or numbers stored as text.
-
Violations of business rules, like a loan amount exceeding the defined maximum limit.
Suppressed Exceptions
Suppressed exceptions are records that fail validation but cannot be corrected or reprocessed due to unresolved logic, missing relationships, or other complexities. These records are not immediately fixed but are logged and set aside, often indefinitely, until someone decides to revisit them or delete them permanently.
Examples of Suppressed Exceptions:
-
Data fields with undefined logic or ambiguous values that do not fit the current business rules.
-
Invalid records with no clear resolution, such as a loan record with incomplete customer information from an external source.
| Aspect | Active Exceptions | Suppressed Exceptions |
|---|---|---|
| Handling | Corrected and reprocessed immediately. | Logged and deferred indefinitely. |
| Priority | High, requires resolution and revalidation. | Low, action is optional or postponed. |
| Examples | Missing fields, format errors, rule breaks. | Undefined relationships and logic. |
| Final Outcome | Reintegrated into the ETL pipeline. | Reviewed later or permanently suppressed. |
Having understood the various tasks that make up the ETL chain, we will next look at how all these tasks can be integrated into a data workflow.