What are the main types of data pipelines?
-
There are different kinds of data pipelines that can be classified into batch processing pipeline and streaming pipeline. Batch pipelines process data in batches at specific periods. Batch pipelines can be used for processing information which do not require to be available at once, for example, processing of daily reports and large data sets. Stream pipelines process data in an endless flow, constantly processing information as it is generated. Stream pipelines are useful for applications that require low-latency information, for example, transaction monitoring, IoT, and real-time analytics. There are different types of data transformation, for example, ETL and ELT. ETL means Extract, Transform, and Load, whereas ELT means Extract, Load, and Transform. In the former case, the data transformation happens before it gets into the target system, while in the latter one, data is loaded first and transformed later.