In recent past , I have come across many discussions where project teams are asking – #databricks and #snowflakes are competitors or complementing each other ?
Well, at this point, they are complementing each other, where data ingestion & processing is taking care by Spark and Data Storage & in-db processing ( ELT) and ACL are taken by snowflake.
But, with huge funding, both enterprises will be competing, primary in datalake and data processing zone.
Spark (3.x) with delta has added many features for datalake or lakehouse and so did snowflake. Which one to choose – depends on your business
1. Spark is developer tool which means it can do very complex transformation but coding need expertise (py,java or scala) – while snowflake is sql based which many not need seasoned developers. Though spark has sparkSQL engine which can transform SQL into code seamlessly.
2. Spark started as scalable ETL tool (in-memory processing ) while snowflake started as elastic cloud DB , separating storage and compute.
3. Spark codes can be easily plugged in data pipeline while snowflake SQL are run inside snowflake cloud only.
I have attached a factsheet but would love to elaborate if someone need to understand more.
viveksingh36 