While data-driven decision making is the number-one indicator of success for today's businesses, a new global study conducted by Dimensional Research for Fivetran, the leading automated data integration provider, shows organizations largely unable to leverage the ever-increasing amount of data they have on hand. According to the study, 44 percent of respondents say that key data is not yet usable for decision making and 68 percent say additional business insights can be extracted from existing data if they only had more time. This is largely attributed to inefficiencies faced by data engineers, who are responsible for transforming data throughout an organization into a format that can be easily analyzed. 

Fivetran is unveiling these and many other findings from an online survey of 543 data professionals across five continents during today's Data Engineer Appreciation Day, a live virtual event that brings together engineers and professionals from around the globe to discuss the state of the industry, share valuable insights, and celebrate data engineers.

Inefficiencies are blamed for data engineers' woes. According to the survey: 

• Nearly all participants (98 percent) have problems building pipelines to access their data.
• 98 percent of participants also say their pipelines break, with 51 percent stating it happens more frequently than once a month.
• Respondents who reported pipelines breaking daily cite source availability and data schema changes as the top reasons.

As a result, business suffers with data engineers spending too much time fixing and maintaining pipelines, rather than focusing on higher-value tasks that generate a positive impact on the business. 66 percent report reduced operational efficiency and 59 percent report delayed decisions or lost opportunities due to broken pipelines.

"Data engineers are the forces behind revenue-impacting business decisions, yet they are still burdened by the limitations of brittle, hard-to-maintain data pipelines. Companies should be aware of the challenges these valuable employees face so they can deliver reliable data more easily to business stakeholders," said George Fraser, CEO of Fivetran. "Fivetran provides reliable access to data through fully managed data pipelines and SQL transformations -- with 99.9 percent guaranteed uptime. This allows companies to put data to work and make decisions faster, instead of spending valuable resources on building and maintaining pipelines themselves."

For large enterprises and small-to-mid size organizations alike, having immediate and reliable access to data is critical for decision-making. The study indicates that pipeline challenges for data engineers continue to strain the process, and businesses lose out on valuable insights as a result.

Additional findings from the survey include:

• 79 percent of companies plan to hire more data engineers within the next year
• 72 percent of organizations move source data daily or more frequently
• 59 percent of companies use 11 or more data sources
• 91 percent say source availability problems are the reason why pipelines break daily
• 55 percent say it takes longer than one business day to repair pipelines
• 86 percent use multiple solutions to build new data pipelines
• 41 percent say a lack of data strategy (policies, processes, standardized technology) is the top data-related challenge their organization faces