Virtualization Technology News and Information
Article
RSS
Astronomer Introduces Astro Observe to Provide Unified Full-Stack Data Orchestration and Observability
Astronomer announced the private preview of Astro Observe. Astro Observe allows any team using Airflow to now have access to an actionable view of the data supply chain and extends Astronomer beyond data orchestration into a platform for data operations. This means proactively identifying opportunities for optimization, zeroing in on problems areas for rapid interventions, and benefitting any team's Airflow projects - regardless of if they are on Astro, open source Airflow or another managed service. By adding observability to the existing data orchestration capabilities of Astro, businesses can now easily build, run, and observe data pipelines as code for seamless data operations. Astro Observe means organizations can now support the development, expansion and upkeep of modern data products better than ever.

"Data products have become just as business-critical as software products, and any failure can have an outsized impact on revenue and reputation," said Julian LaNeve, CTO, Astronomer. "Astro Observe unifies orchestration and observability across the full data stack on a single platform, meaning the reliability and trust of data products are improved, development velocity is increased, costs are lowered, mission-critical SLAs are respected and critical data assets are better secured. We want to give data teams the same operational rigor as software teams."

Delivering reliable data products is imperative, but complex

Reliance on data has evolved over the past decade from powering basic internal dashboards to data becoming a product itself, with source data packaged alongside artifacts like metadata, transformation logic, docs and tests, access policies-all exposed as APIs to drive a diverse set of mission-critical use cases that depend on data orchestration to work. Whether it's retail recommendations with dynamic pricing, automated customer support, financial trading strategy, regulatory reporting, or AI-full stack data orchestration has become the linchpin of modern software solutions.

Observability and Orchestration on a single platform provides connective tissue between the entire data stack

Observability has historically already been a differentiator for Astronomer compared to OSS Airflow and other managed services, with logs and metrics to provide operational visibility across all of a customer's Airflow deployments and workspaces. Astro Observe introduces data observability as an additional capability of our platform. Central to Astronomer's data observability capabilities is the use of OpenLineage as the framework to make data pipelines observable by automatically collecting and correlating detailed information about their behavior and data movement. In the last few years, Astronomer has invested heavily in the development of OpenLineage and it has emerged as an industry standard for solving problems around data observability.

Key features and benefits of Astro Observe

  • SLA Dashboard: Enables data product owners to be assigned, to provide a clear picture of accountability and ownership. Owners of data products can set policies and monitor thresholds for data freshness and delivery times to better align with business outcomes.
  • Dependency Graph: Visibility into upstream and downstream relationships across data products, providing critical context for issue resolution and ensuring compliance.
  • Recommendation Engine: Leverages best practices from Airflow alongside a holistic view of deployments to surface actionable recommendations for optimizing trust and reliability of data products.
  • Predictive Alerting: Allows teams to pinpoint and understand risk before the delivery of data products are impacted; providing time and context necessary to remediate and make adjustments across the stack to ensure the on-time delivery of fresh and reliable data.
  • Single Destination for Orchestration and Observability: Astronomer provides a single destination for teams to orchestrate and manage pipelines along with the visibility they need to monitor performance and make adjustments as needed to solve problems or make proactive optimizations; minimizing context switching, and allowing for one interface to understand and take action. As modern data stacks become more complex with dependencies that sprawl across multiple platforms, teams, and AIrflow deployments, Astronomer's centralized view helps teams to quickly assess and make decisions about the management of their projects.

To learn more, visit the resources below to get started with Astro today.

Published Tuesday, September 10, 2024 2:06 PM by David Marshall
Filed under:
Comments
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
Calendar
<September 2024>
SuMoTuWeThFrSa
25262728293031
1234567
891011121314
15161718192021
22232425262728
293012345