What is data engineering?


Data Engineering

Data engineering uses data as the means for understanding a process. For a more comprehensive introduction, see our White Paper on Data Engineering.

The data might be generated in many ways, or subset of the available data may be used. Data engineering uses data analysis techniques from statistics, machine learning, pattern recognition or neural networks, together with other technologies such as visualization, optimization, database systems, prototyping tools and knowledge elicitation. The goal is to use the available data or generate more data, and to thereby understand the process being investigated. The process of analysing the data, creating new analysis tools specifically for the task, and working with the domain experts is a key aspect of this engineering task. We will be using Bayesian data analysis methods (which occur throughout the different communities).

You may have also head of the terms data mining and knowledge discovery, exploratory data analysis, intelligent data analysis, and so forth. These are similar.


wray@ic.eecs.berkeley.edu