I discussed all the option for memory, CPU and storage optimization in all stages of data pipeline and suggested in details what are all the parameters that needs to be tuned to optimize the data pipeline with examples.