v5.5.0
Date of Release: September 2023
This section consists of the new features and enhancements introduced in this release. |
- Infoworks now supports EMR version 6.6 in addition to 6.2. For more information, refer to the "Type and Run Time Version" field in the Compute section. Also, note that support for 5.32 has been deprecated.
- Infoworks will support creation of multiple profiles in single snowflake environment. Users using snowflake environment will now have a choice to select profile during source, pipeline and pipeline group creation. For more details, please refer to the Table Schema section here.
- For sources created on Snowflake environments, time data type has been added. Additionally there is a textbox which takes the precision for time data type columns. The value of this is prefilled during metacrawl but it can be edited. Refer to Configuring a Table.
- Infoworks now supports access control when using Databricks clusters and jobs. Infoworks will accept access control field as json string and use the field as is in Databricks API calls for cluster creation and job submission. Access control must be enabled and configured as per Databricks documentation before usage in infoworks. For detailed information, please refer to the Access Control List section here.
- Now the user will have the option to synchronize pipeline export table schema with source table schema for the targets: BiqQuery Target Table, Snowflake Target Table, Postgres Target Table, Teradata Target Table, Oracle Target Table, Redshift Target Table, SQL Server Target Table, Azure Synapse Target Table. To know more, refer to the sections listed under Pipeline Targets.
- Support for storing and accessing client secret of service principal from Azure Key Vault.Now we can have two sets of service principal one for Azure key vault authentication and other for authentication of Databricks api whose client secret stored in azure key vault. Refer to Managing Secrets.
- With SLA configurations, user can now configure expected run time of job (eg: hours and minutes) on entity level (table_groups, pipeline, pipeline_groups, workflow) and get notified if job run exceeds configured run time. For more details, view pages: SLA Configuration and Criticality and Using Operations Analyst Dashboard.
- Now the SQL Import feature supports the DELETE and UPDATE queries. Refer to Importing SQL.
- Onboarding flow for SFDC sources modified to enable the ability to only onboard the required tables to the source. For more information, view Onboarding Data from SalesForce page.
- New read only role added to provide view only access of infoworks. For more details, check out the Editing a User section here.
- Infoworks now supports native idle timeout option for Dataproc and EMR persistent clusters (previously only available for Databricks).
- There are a few changes in the transitive access control of domains through the “Accessible Domains” functionality. Please review here for specific changes.
- The Databricks runtime version for 9.1 has been deprecated. More information can be found under the Compute section.
- There are changes in SAML ACS URL format to work with V3 APIs. Please check here for information.
This section consists of the resolved issues in this release:
| JIRA ID | Issue |
|---|---|
| IPD-22036 | Pipeline build fails when the external target BQ table already exists and is clustered/partitioned. |
| IPD-22090 | Export to delimited files: Timestamp Format issue. |
| IPD-22351 | Storage project requires create permissions despite parent project feature enablement. |
| IPD-22615 | Partition and the clustering details are missing in the big query table created by infoworks pipeline. |
| IPD-22721 | Advance configuration for streaming_group_id_prefix not working as expected. |
| IPD-22025 | SQL Pipeline: Sql pipeline validation API is not filtering the advanced configuration by "IsActive" flag. |
| IPD-22034 | Non Visual Infoworks SQL Pipeline cannot execute SnowSQL. |
| IPD-22037 | User managed table options are not available for streaming sources in CDW environment. |
| IPD-22048 | Dropdown option is not selectable in Interactive Compute Mappings in workflow migration. |
| IPD-22076 | Sync to external target to delimited files is showing "Please select a Data Connection" on saving the configuration. |
| IPD-22332 | Unable to set-active pipeline API in v5.4.2. |
| IPD-22421 | Workflow config migration API is not behaving as expected in Infoworks 5.4.2. |
| IPD-21534 | Initialize & ingest/truncate job is not resetting/updating the value of key last_merged_watermark. |
| IPD-22113 | v5.4.1.3 - Pipeline creation does not happen with compute template name. |
| IPD-22817 | Need validation on batch_engine key during the pipeline creation in API flow. |
| IPD-22755 | GET and PUT methods aren't returning table doc with user_managed_target_path key for user managed tables. |
| IPD-22857 | Segment load using fetch type as TPT and with multi-water mark column is running for 18 hours in the environment. |
| IPD-22963 | The second pipeline build is causing duplicate records when the load incrementally is enabled and the sync type is set to append in 5.3.X and 5.4.X versions. |
| IPD-22964 | Kafka schema registry password emptied post upgrade to 5.4.1 from 5.3.0.x. |
| IPD-22983 | Incremental pipeline bringing entire dataset when the source table is from confluent source with incremental mode as append. |
| IPD-22985 | Append and merge mode tables getting overwritten in each run of confluent source ingestion. |
| IPD-22038 | The ingestion job continued to stay in the blocked state even if we unlock them on the admin page. |
| IPD-23007 | Project ID field issue on the csv source configured on the BigQuery data environment. |
| IPD-22972 | Azure Key Vault drop down doesn't show context of the secret for non-Admin users. |
| IPD-23079 | Sync to target job for Postgres fails with authentication error. |
| IPD-23080 | Job logs show password for Sync to Target Job. |
| IPD-23128 | Unsaved changes popup when updated table configurations via API in v5.4.2. |
| IPD-23129 | For SCD 2 type, few fields are not getting saved in Table Document in v5.4.2. |
| IPD-23228 | Blob storage changes for supporting 17.2 TPT. |
| IPD-23239 | Create source API on snowflake meta sync source is failing. |
| IPD-23264 | Relax scale and precision check in snowflake onboarding. |
| IPD-23197 | Backend Fix (Ingestion) - UI/RESTAPI behaviour inconsistency while updating Sync to Target properties. |
| IPD-23200 | GET source configuration call is failing on snowflake meta sync source. |
| IPD-23263 | IWX scheduler issue - 5.3.0.14. |
| IPD-23418 | BigQuery pipelines are failing with java.lang.IllegalArgumentException: Provided query is null or empty error after the upgrade. |
| IPD-23327 | Inactive spark advanced configs are taking effect in 5.4.1.4. |
| IPD-23452 | Rest API - Bug in 5.4.1 config migration where few keys are in camelCase (should be in snake_case). |
| IPD-23498 | Project ID field issue on the teradata source configured on the BigQuery data environment. |
| IPD-23515 | Ingestion job on BQ environment is running queries on GCP project present in service account JSON instead of parent_project. |
| IPD-23526 | Issues while using partitioned externally created BQ table in our pipelines. |
| IPD-23613 | Last Modified Date in Workflow is shown incorrectly. |
| IPD-23663 | File Archival process is not archiving files with only header records post upgrade to 5.4. |
| IPD-23636 | Pipeline Jobs list not visible. |
| IPD-23647 | The staging table created by Infoworks during pipeline build will be persisted in the BigQuery if the load job fails. |
| IPD-23754 | Records were missing in the BigQuery target table even though the pipeline was completed successfully.. |
| IPD-23822 | Schema name is null/empty for incremental pipelines in BQ environment on using use_ingestion_time_as_watermark key |
| IPD-23719 | Pipeline versions getting emptied on production. |
| IPD-23313 | Environment details API throws authentication error for prodops user, however the same can be viewed from UI by prodops user |
| IPD-23589 | Issues when selecting all tables in UI for onboarding, metadata crawl. |
| IPD-23634 | List table APIs bringing duplicates during different offsets parameters in v5.4.2.3. |
| IPD-23714 | Source connection details API not returning the additional connection parameters in 5.4.x. |
| IPD-23716 | Add File mapping API failing, causing the mass ingestion script failure. |
| IPD-23733 | "Make available in Infoworks domains" not coming in source details API. |
| IPD-23734 | Source Config migration API on changing the table group names in iw_mappings not reflecting after import. |
| IPD-23783 | Source config migration API doesn't return use_staging_schema_for_infoworks_managed_tables |
| IPD-23899 | Refresh token not generated for newly created users in 5.4.1.6. |
| IPD-23949 | Source Config Migration API on 5.4.2.4 latest build started to fail. |
| IPD-23963 | Issues with table-level email notification. |
| IPD-23732 | Add tables api does not return table_type in its json dump. |
| IPD-24007 | View run is not loading workflow run page beyond recent 20 workflow runs limit. |
| IPD-24164 | Databricks Environment - switch to global init scripts. |
| IPD-24213 | Issues with downloading Databricks log file. |
| IPD-24406 | Metadata crawl for Confluent Kafka source is failing with no messages found for Topic. |
The following section contains known issues that Infoworks is aware of, and is working on a fix in an upcoming release:
| JIRA ID | Issue |
|---|---|
| IPD-23583 | Pipeline that is a part of pipeline group can be deleted. Ideally, this operation should not be allowed. |
| IPD-23078 | Table preview request times out for the first time for SQL source. |
- Persistent Cluster : Cluster restart does not work reliably with inactivity-timeout set less than 10 minutes.
- For SQL pipelines, use of snowflake variables as a table identifier / table name not supported.
- BigQuery as a sync to target is not supported on EMR 6.6.
- Deploy mode as a Cluster is not supported on EMR and Dataproc.
For Kubernetes-based installation, refer to Infoworks Installation on Azure Kubernetes Service (AKS).
For VM-based installation, refer to VM-Based Deployment.
For more information, contact support@infoworks.io.
For upgrading from 5.4.2.x to 5.5.0 for Azure Kubernetes, refer to Upgrading Infoworks from 5.4.2.x to 5.5.0 for Azure Kubernetes.
For upgrading from 5.0 to 5.5.0 for VM, refer to Upgrading Infoworks from 5.0 to 5.5.0 for VM.
Was this page helpful?