Optimizing Spark in the Cloud - with Jean-Yves Stephan

Building the Backend: Data Solutions that Power Leading Organizations

This episode features Jean-Yves Stephan Co-Founder & CEO @ Data Mechanics (recently Acq. by Spot by NetApp), during our discussion we talk about optimizing Spark to run in the cloud at a low cost.

Top 3 Value Bombs:

• Running Spark CAN be expensive but there are ways to reduce your current operating costs by 50-75% by smart automations (i.e. tune for node type, memory and CPU).
• Spot instances can lower your costs by utilizing unused instances.
• Creating serverless architectures and using containers will allow for more flexibility with deployment models and scalability.