Releases: open-metadata/OpenMetadata
OpenMetadata 0.11.2-release
What's Changed
- Cherry pick 02d3695 by @Sachin-chaurasiya in #6007
- UI cherry pick by @Sachin-chaurasiya in #6182
- Fix #6061 Backend: Custom properties creation error (#6098) by @Sachin-chaurasiya in #6194
- Fix #6179 Edge not getting generated when multiple tasks in pipeline(#6181) by @Sachin-chaurasiya in #6195
Full Changelog: 0.11.1-release...0.11.2-release
Contributors
Assets 5
OpenMetadata 0.11.1-Release
What's Changed
- Prepare postgres migrations by @pmbrull in #5814
- Fixed: Team & Users page is stuck at loading on trying to join the team by @ShaileshParmar11 in #5816
- Fix typo by @eitsupi in #5818
- Defined a constant instead of duplicating string literal #1982 by @truemanD in #5819
- Refactor code to consolidate common code to EntityRepository and clea… by @sureshms in #5824
- Fix #5826 UI : Add Error Boundaries by @Sachin-chaurasiya in #5827
- Fix #5532 Suppress Yaml Warnings for generate typescript command by @vivekratnavel in #5783
- Fixes #5558 Add support for enumerating Operations and Resources for authoring policies by @sureshms in #5561
- Added event error and fernet classes to openmetadata-core by @parthp2107 in #5799
- Fix#5843 UI : Do not show the request description icon in entities that has no support for tasks yet by @Sachin-chaurasiya in #5844
- added condition to echo only with debug flag in bootstrap.sh by @parthp2107 in #5845
- Cypress: Cover service page issue related to description by @ShaileshParmar11 in #5529
- [Fix] Removed volume instructions from dockerfile. by @Rabi-Sahoo in #5851
- chore(ui): update types with respective libs instead hardcoded by @chirag-madlani in #5848
- Fixes #5245 Entity tag is not getting removed, even after glossary te… by @sureshms in #5838
- Fix #5471 Creating a tag with an illegal character prevents utilizing the tagging system until manually removed by @Sachin-chaurasiya in #5854
- Fix: issue 5645: UI feedback 0.11 by @ShaileshParmar11 in #5853
- Fixed Column level lineage issue #5825 by @ShaileshParmar11 in #5861
- fix(ui): #5836 User profile should show user related activity only by @chirag-madlani in #5864
- Fix #2895: Added Foreign Key support by @ulixius9 in #5856
- added config to publish to maven repository in catalog pom.xml by @parthp2107 in #5870
- Fixes #3133 -- Adding Additional Column Tests by @TeddyCr in #5867
- Fix #5871 - Validate rest and update FQN for DatabaseSchema by @pmbrull in #5873
- Feat: UI button to pause/unpause ingestion pipeline executions by @ShaileshParmar11 in #5884
- issue-5757: s3-prefix-fix-added by @codingwithabhi in #5876
- fix(ui): #5850 loading issue for teams and user page by @chirag-madlani in #5881
- Feat: Added Lineage support for topic and mlModal by @ShaileshParmar11 in #5888
- Fixes #5866 -- Implement Support for Median Metric by @TeddyCr in #5887
- Fix #5764 Backend: Add support for filter by task status by @vivekratnavel in #5899
- Fix #5857 Add icon for foreign key constraint by @Sachin-chaurasiya in #5904
- UI: Fix browser zoom breaks UI layout by @vivekratnavel in #5909
- revert(ui): count badge was limit to 999 by @chirag-madlani in #5910
- test(ui): unit tests added for profilerUtils by @chirag-madlani in #5922
- Update docs recipe by @pmbrull in #5886
- Dbt Fixes and removed urllib from DBT and ometa by @OnkarVO7 in #5928
- fixed issue: Error handling not present for add/delete/update Webhook by @ShaileshParmar11 in #5911
- ✨ UI: Display a switch to show Closed Tasks vs Open Tasks by @Sachin-chaurasiya in #5929
- Update python CI order by @pmbrull in #5934
- fix(ui): owner search issue and show total count on load by @chirag-madlani in #5932
- Fix metadata es DAG by @pmbrull in #5933
- Fix Authentication Public Keys env variable in docker compose by @vivekratnavel in #5948
- Fix #5877: Renamed Database to DatabaseSchema for Oracle Connection by @ulixius9 in #5889
- Fix #5905: Bootsrap Migration Fix by @ulixius9 in #5930
- Fix #5926 Missing downstream in pipeline task view by @Sachin-chaurasiya in #5950
- Fix inputvariables by @Vj-L in #5961
- Prepare OpenMetadata Main for Next Release by @Vj-L in #5962
- feat(ui): refresh token support for OIDC client login by @chirag-madlani in #5923
- Fix #5809 Show count of schemas and tables being deleted when a database service is being deleted. by @Sachin-chaurasiya in #5963
- Fixes #5964 -- Trino Median function and columnValuesToMatchRegex wording by @TeddyCr in #5965
- Update missing FQN by @pmbrull in #5966
- Fix issue 5955 non-joinable team also visible at the time of signup by @ShaileshParmar11 in #5957
- Fix #5956 Long team names go out of the box by @Sachin-chaurasiya in #5959
- Fix #5968 Airbyte logo is missing by @Sachin-chaurasiya in #5970
- Connection options fix by @ulixius9 in #5971
- Snowflake View Definition Fix by @ulixius9 in #5974
- Docs - Migrate overview and restructure menu by @pmbrull in #5983
- Fixed #2188: Can add and remove more than 10 tags at once. by @parthp2107 in #5040
- Docs migration - Deployment and upgrades by @pmbrull in #5984
- Fix airflow docs by @pmbrull in #5985
- Fix #5092: Allow tag names to be 64 chars in length by @harshach in #5986
- Fixed issue-5897: UI: Tags usage link is broken by @ShaileshParmar11 in #5975
- Fix #5860: Dashboard usage throws Entity type not supported by @harshach in #5987
- Fixed issue 5981 UI: Add Team button doesn't work in the users page by @ShaileshParmar11 in #5992
- Snowflake Password Optional Field by @ulixius9 in #5994
- Fix UI: User name overflows out of the profile dropdown by @Sachin-chaurasiya in #5993
New Contributors
Full Changelog: 0.11.0-release...0.11.1-release
Contributors
Assets 5
OpenMetadata 0.11.0-Release
Data Collaboration - Tasks and Emojis
Data Collaboration has been the prime focus of the 0.11 Release, the groundwork for which has been laid in the past several releases. In the 0.9 release, we introduced Activity Feeds, Conversation Threads, and the ability to request descriptions. In this release, we’ve added Tasks, as an extension to the ability to create conversations and post replies.
We are particularly excited about the ability to suggest tasks. This brings the collaboration to the next level where an organization can crowdsource the knowledge and continuously improve descriptions.
Column Level Lineage
#2931
In OpenMetadata, we primarily compute column-level lineage through SQL query analysis. Lineage information is consolidated from various sources, such as ETL pipelines, DBT, query analysis, and so on. In the backend, we’ve added column-level lineage API support. The UI now supports exploring this rich column-level lineage for understanding the relationship between tables and performing impact analysis. While exploring the lineage, users can manually edit both the table and column level lineage to capture any information that is not automatically surfaced.
Custom Properties
The key goal of the OpenMetadata project is to define Open Metadata Standards to make metadata centralized, easily shareable, and make tool interoperability easier. We take a schema-first approach for strongly typed metadata types and entities modeled using JSON schema as follows:
OpenMetadata now supports adding new types and extending entities when organizations need to capture custom metadata. New types and custom fields can be added to entities either using API or in OpenMetadata UI. This extensibility is based on JSON schema and hence has all the benefits of strong typing, rich constraints, documentation, and automatic validation similar to the core OpenMetadata schemas.
Advanced Search
Users can search by multiple parameters to narrow down the search results. Separate advanced search options are available for Tables, Topics, Dashboards, Pipelines, and ML Models. All these entities are searchable by common search options such as Owner, Tag, and Service.
Glossary UI Updates
The Glossary UI has been upgraded. However, the existing glossary functionality remains the same, with the ability to add Glossary, Terms, Tags, Descriptions, Reviewers etc... On the UI, the arrangement displaying the Summary, Related Terms, Synonyms, and References has been changed. The Reviewers are shown on the right panel with an option to add or remove existing reviewers.
Profiler and Data Quality Improvements
Profiling data and communicating quality across the organization is core to OpenMetadata. While numerous tools exist, they are often isolated and require users to navigate multiple interfaces. In OpenMetadata, these tests and data profiles are displayed alongside your assets (tables, views) and allow you to get a 360-degree view of your data.
Great Expectations Integration
While OpenMetadata allows you to set up and run data quality tests directly from the UI, we understand certain organizations already have their own data quality tool. That’s why we have developed a direct integration between Great Expectations and OpenMetadata. Using our openmetadata-ingestion[great-expectations] python submodule, you can now add custom actions to your Great Expectations checkpoints file that will automatically ingest your data quality test results into OpenMetadata at the end of your checkpoint file run.
ML Models
In this release, we are happy to share the addition of ML Model Entities to the UI. This will allow users to describe, and share models and their features as any other data asset. The UI support also includes the ingestion through the UI from MLflow. In future releases, we will add connectors to other popular ML platforms.
This is just the beginning. We want to learn about the use cases from the community and connect with people that want to help us shape the vision and roadmap. Do not hesitate to reach out!
Connectors
In every release, OpenMetadata has maintained its focus on adding new connectors. In the 0.11 release, five new connectors have been added - Airbyte, Mode, AWS Data Lake, Google Cloud Data Lake, and Apache Pinot.
OpenMetadata 0.10.4-release
OpenMetadata 0.10.3-release
OpenMetadata 0.10.2-release
OpenMetadata 0.10.1-release
What's Changed
- Support for Postgres as OpenMetadata Store #4601
- UI Improvements in 0.10.1 Release #4600
- Support JWT Token Generation for Bot Accounts #4637
- UI Ingestion Improvements - Support for Dashboards & Messaging Services #4843
- Security: Fix Azure SSO and support refresh tokens (#4988) by @vivekratnavel in #4989
New Contributors
- @cometta made their first contribution in #2916
- @snyk-bot made their first contribution in #3130
- @Beetelbrox made their first contribution in #3185
- @sfirke made their first contribution in #3361
- @mosiac1 made their first contribution in #3599
- @troyel made their first contribution in #3695
- @dvecerdea made their first contribution in #3800
- @Curiosady made their first contribution in #3892
- @barloc made their first contribution in #4037
- @Vj-L made their first contribution in #4092
- @TeraInferno made their first contribution in #3720
- @vitoravancini made their first contribution in #4733
- @habibrhmn made their first contribution in #4800
- @filip-xmargin made their first contribution in #4926
- @mohitdeuex made their first contribution in #4956
Full Changelog: 0.10.0-release...0.10.1-release
Contributors
Assets 5
OpenMetadata 0.10.0-release
What's Changed
Support for Database Schema
OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. For each external data source, we ingest the database, as well as the tables that are contained underneath the schemas.
Support for Hard Delete
OpenMetadata supported soft deletions. Now, we also support the hard deletion of entities through the UI, APIs, and ingestion. Hard deleting an entity removes the entity and all of its relationships. This will also generate a change event.
Deploy Ingestion from UI
OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework as well as the UI. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.
Download DBT Manifest Files from Amazon S3 or Google Cloud Storage
Previously, when ingesting the models and lineage from DBT, we passed the path of the DBT manifest and catalog files directly into the workflow. We’ve worked on improving the quality of life of DBT. Now, we can dynamically download these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the DBT, extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the metadata extraction to be able to ingest metadata.
JSON Schema based Connection Definition
Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each connector, we’ve worked on the full refactoring of the ingestion framework. We now use the pydantic models automatically generated from the JSON schemas for the connection definition.
Airflow Rest APIs
The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API directly on top of Airflow using plugins. This passes the connection information to automatically generate all the dags and prepares handy methods to help us test the connection to the source before creating the service.
UI Changes
- The UI improvements are directed towards providing a consistent user experience.
- Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. This generates an ‘EntityDeleted’ change event.
- Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.
- UI Support for Database Schema as part of FQN: The database schema has been introduced in the 0.10 release. All the entity pages now support Database Schema in the UI.
- Lineage Editor: Improvements have been made to the lineage editor.
- Teams: While signing up in OpenMetadata, the teams with restricted access are hidden and only the joinable teams are displayed.
- Team Owner: An Owner field has been added to the Team entity. Only team owners can update the teams.
- Activity Feeds: The Activity Feeds UI supports infinite scrolling.
- Add User: A user can be added from the Users page.
Security Changes
Support Refresh Tokens for Auth0 and Okta SSO
The JWT tokens generated by the SSO providers expire by default in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 and Okta SSO. The tokens are refreshed silently behind the scenes to provide an uninterrupted user experience. In the future releases, we’ll continue to stabilize authentication and add refresh tokens for the other SSO providers.
Custom OIDC SSO
OpenMetadata now supports integration with your custom built OIDC SSO for authentication. This is supported both on the front end for user authentication as well as on the ingestion side.
Azure SSO
Support has been added for Azure SSO on Airflow.
New Contributors
- @cometta made their first contribution in #2916
- @Beetelbrox made their first contribution in #3185
- @sfirke made their first contribution in #3361
- @OnkarVO7 made their first contribution in #3408
- @mosiac1 made their first contribution in #3599
- @troyel made their first contribution in #3695
- @dvecerdea made their first contribution in #3800
- @Curiosady made their first contribution in #3892
- @MilanBariya made their first contribution in #3949
- @barloc made their first contribution in #4037
- @Vj-L made their first contribution in #4092
- @TeraInferno made their first contribution in #3720
Full Changelog: 0.8.1-release...0.10.0-release
Contributors
Assets 5
OpenMetadata 0.9.1-release
Collaboration
- Conversations in the main feed
- Users can ask each other questions, add suggestions and replies
- Table details - Click through on usage to see who or what services are using it, and what queries are pulling from it.
Data Quality
- Ability to create and monitor the test cases
- Data Quality Tests support with Json Schemas and APIs
- UI Integration to enable users to write tests and run them on Airflow
Glossary
- Glossaries are a Controlled Vocabulary in an organization used to define the concepts and terminologies specific to a particular domain.
- API & Schemas to support Glossary
- UI support to add Glossary and Glossary Terms.
- Support for using Glossary terms to annotate Entities and Search using Glossary Terms
Connectors
- Apache Iceberg
- Azure SQL
- Clickhouse
- Clickhouse Usage
- Databricks
- Databricks Usage
- Delta Lake
- DynamoDB
- IBM DB2
- Power BI
- MSSQL Usage
- SingleStore
- Apache Atlas, Import Metadata from Apache Atlas into OpenMetadata
- Amundsen, Import Metadata from Amundsen into OpenMetadata
Lineage
- DataSource SQL Parsing support to extract Lineage
- View Lineage support
Pipeline
- Capture pipeline status as it happens
Security
- Security policies through the UI
- Configuration personas and authorization based on policies
- AWS SSO support
New Contributors
- @cometta made their first contribution in #2916
- @Beetelbrox made their first contribution in #3185
- @sfirke made their first contribution in #3361
Full Changelog: 0.8.1-release...0.9.1-release
Contributors
Assets 3
OpenMetadata 0.9.0-release
What's Changed
- Handle description and tag updates NPE by @harshach in #2856
- Generating Typescript Types from Json Schema. by @Sachin-chaurasiya in #2851
- Fix #2835, #2836 Lineage entities overlap and Descriptions in Lineage view do not have formatting. by @Sachin-chaurasiya in #2840
- Fix #2778 UI : Add Queries tab for table details page. by @Sachin-chaurasiya in #2793
- Fix: #2521 - Add owner support for all services and database by @darth-coder00 in #2858
- Tableau Lineage Implementation by @ayush-shah in #2850
- Table Constraints Added - Ingestion by @ayush-shah in #2854
- Add support for default role (#2631) by @mithmatt in #2676
- Fix #2837: Add a Feed Count API with entityLink filter support by @vivekratnavel in #2860
- Fix #2845 - Init Profiler Workflow by @pmbrull in #2862
- Fix access control rule userRoleAttr to point to roleName by @mithmatt in #2644
- Support swapping default roles by @mithmatt in #2868
- Extend team to support defaultRoles by @mithmatt in #2885
- fix: cherry pick for 0.8.3 by @akash-jain-10 in #2893
- fix #2869: snowflake-usage processor fixed by @ulixius9 in #2896
- Fix #2902 - Check for empty payload by @pmbrull in #2903
- Fix #2867: Add support for GET /feed/{id} by @vivekratnavel in #2892
- Fix #2894 - Profiler Processor & Metrics by @pmbrull in #2900
- Fix : Roles page is showing blank UI when switching to the users tab. by @Sachin-chaurasiya in #2922
- fix incorrect query by @cometta in #2916
- Fix #2890 Show columns icons in Lineage UI enhancement. by @Sachin-chaurasiya in #2924
- Fix #2901: fixed error message by @ulixius9 in #2917
- Removed ColumnDataTypeDisplay check with ColumnDataType by @ayush-shah in #2928
- Fix #2871: Make search magnifying glass clickable to search by @Sachin-chaurasiya in #2923
- Fixes #2899 - Not getting proper relation between parent and child gl… by @sureshms in #2941
- Fix #2953: NoopAuthorizer should create a anonymous user by @harshach in #2940
- Changing selenium version by @kshinde2512 in #2934
- Added Error Handling in Policy tags - Bigquery by @ayush-shah in #2933
- Fix #2905: Added MSSQL Usage Connector by @ulixius9 in #2948
- Refactor EntityReference list population with id list for a given entity type by @mithmatt in #2956
- Fix #2865: Thread should have its own message and posts should only be the replies by @vivekratnavel in #2944
- Support returning teams for GET /roles by @mithmatt in #2958
- Fixes #2942 - Fix bugs reported by Sonar issue on release 0.9 by @sureshms in #2943
- Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics by @pmbrull in #2926
- Bigquery lineage mapping by @codingwithabhi in #2849
- Fix #2897 - Profiler CLI by @pmbrull in #2967
- Fix #2873: DataQuality test specifications and APIs by @harshach in #2906
- fix: Bump up release version 0.8.4 for docker compose by @akash-jain-10 in #2968
- Fix #2539: redshift lineage mapping by @ulixius9 in #2828
- Adding Webhooks selenium tests by @kshinde2512 in #2911
- Assign default roles from Teams to User by @mithmatt in #2947
- Fix #2966: JWT filter should compare times in UTC to check the validity by @vivekratnavel in #2971
- Update all *.java in openmetadata-ui to use lombok.LOG by @mithmatt in #2973
- Use RequiredArgsConstructor in openmetadata-ui/*.java by @mithmatt in #2974
- fix #2809: Auto populate View Lineage to Tables by @ulixius9 in #2977
- Fix #2965 - Additional Properties set to False by @pmbrull in #2976
- Use lombok Getter to simplify openmetadata-ui/*.java files by @mithmatt in #2975
- Fix #2981 - Update Profile to match TableProfile by @pmbrull in #2982
- Fix #2991: Add ES support for glossary by @harshach in #2993
- Fixes #2962 - Clean up fields parameter choices in GET APIs by @sureshms in #2963
- Fix #2946: Presto Ingestion issues fix by @ulixius9 in #2986
- Fix #2937: added clickhouse usage connector by @ulixius9 in #2987
- Fix Okta Authentication and Validation - Ingestion by @ayush-shah in #2955
- Fix #2996: ChangeEventHandler throws NPE at times by @vivekratnavel in #2998
- Metabase lineage by @codingwithabhi in #3003
- Fix #2834: Merge the change events 'delete' and 'add' as 'update' for Activity Feed by @vivekratnavel in #3006
- updated chrome drivers and added options for headless chrome by @parthp2107 in #3009
- Fix #2807: Merge openmetadata.yaml and openmetadata-server.yaml by @harshach in #2808
- Fix #3012 - Dependency breaking Airflow Container by @pmbrull in #3015
- Fix: #2938 Remove serviceType from service UI route by @Sachin-chaurasiya in #2939
- Feat collaboration by @Sachin-chaurasiya in #2859
- Fix #2984: added azure sso auth by @ulixius9 in #3002
- Fixes #3021 - Restore a soft-deleted container entity along with the … by @sureshms in #3022
- Fixes #3007 - Clean up Sonar flagged issues by @sureshms in #3008
- Fix #3023: Keep ElasticSearch version to 7.13 by @harshach in #3024
- Power BI connector by @ayush-shah in #3019
- Feat : Display a chat icon next to data entities that have threads by @Sachin-chaurasiya in #3026
- Atlas connector by @codingwithabhi in #2855
- ES index: ignore any errors during the es index by @harshach in #1220
- WIP - Selenium fix by @parthp2107 in #3014
- Fix #3037: metadata --version doesn't work by @amiorin in #3038
- Fix #3040 - py_format recipe includes more paths by @pmbrull in #3043
- Fix: #2491 - improvements in google, okta and azure auth by @darth-coder00 in #2929
- fix #3035: handled none entity by @ulixius9 in #3036
- Add UpdateTeam permission support by @mithmatt in #2994
- Fix #3041: Handled AttributeError by @ulixius9 in #3042
- Fix: azure auth callback from UI by @darth-coder00 in #3048
- Fix #2989: Add delete apis for tests by @harshach in #3020
- Fix #3047: OpenMetadata Server fails to run when updating config from Environment variable by @harshach in #3049
- Fix #2866: Add support for PATCH to feed API to be able to resolve a thread by @vivekratnavel in #3027
*...
