Answer selected: Terminology clarification in Spark

Wed, 17 Feb 2021 16:04:05 +0000

The fact is the engine is still the same, regardless of which interface language you use. For some tasks, such as special cleaning we probably do not have SQL commands, and we have to use Scala or Python. Using Zeppelin, you can switch back and forth among languages the engine supports, however it is not a common practice. For some specific tasks, you can use pure Spark SQL or if you want to use the SQL in pyspark or scala, there are functions that can help you achieve the goal.

I believe observing more examples will help you understand when you can use what.

Answered: Logstash takes more than 48 hours to ingest 10 GB CSV file. Is there a way to expedite?

Mon, 15 Oct 2018 02:05:16 +0000

It depends on your system processing power and the way you configure Logstash. If you make more outputs streams it will take longer. Please take a look at this link to see how you can tune the configurations. More than that, you can see some nice tips here.

Ask Ghassem - Recent activity in Big Data Tools

Answer selected: Terminology clarification in Spark

Answered: Logstash takes more than 48 hours to ingest 10 GB CSV file. Is there a way to expedite?