Apache Spark development to sync data into elastic search from multiple table from postgreSql

Virtual Dataset syncing through Spark.

Spark job to written in Scala/Java which will retrieve the data from multiple tables with complex join conditions. There will be multiple stages before the final outcome and each stage can transformed and will be input to next stage for joining with other table/previous output.

Request will have all the metadata in JSON format for the stages involved in the virtual dataset and there detailed joins as well as transformation conditions.

Screenshots attached for more understanding.