Snowpark for Python
Snowflake · Advanced Features
Theory
Snowpark is a set of libraries for Python, Java and Scala that let you work with Snowflake data through a DataFrame API. When you write session.table("orders").filter(col("amount") > 500).group_by("region").agg(sum_("amount")), nothing runs at first; Snowpark builds a plan and, when you call an action such as collect() or save_as_table(), turns it into SQL that runs on a Snowflake warehouse. Data stays in Snowflake instead of being pulled to your laptop.
For custom logic, you can package Python code as UDFs or stored procedures that run in a secure sandbox on the warehouse, using third-party packages that Snowflake makes available. Snowpark-optimized warehouses offer more memory per node for heavy workloads like model training.
Why we use it
Moving large data out of the warehouse to Spark or pandas is slow, costly and creates security copies. Snowpark brings the code to the data, so developers keep Python while Snowflake does the heavy lifting and governance stays in one place.
When to use it
Use Snowpark when transformations are easier in Python than SQL, for ML feature engineering, or to reuse Python libraries. Use plain SQL for straightforward transformations, since Snowpark DataFrames become SQL anyway.
Why Run Code Inside Snowflake
Data never leaves Snowflake's governed environment, so the same RBAC, masking policies, and audit logging that protect SQL queries also protect Snowpark code — a common requirement in regulated industries.
DataFrame API
Snowpark's DataFrame API mirrors pandas/PySpark syntax (.filter(), .group_by(), .select()) but Snowflake translates those operations into SQL under the hood and runs them on a virtual warehouse.
Snowflake ML
Snowflake ML (the snowflake-ml-python library, formerly called Snowpark ML) adds feature engineering, model training, a Feature Store and a Model Registry, so the whole ML lifecycle can run without exporting data to an external platform.
How it works
- Write DataFrame code in Python
- Snowpark builds a lazy plan
- Is an action like collect() or save_as_table() called? Yes: Generate SQL and send it to Snowflake. No: Keep building the plan.
- Warehouse executes the SQL and any Python UDFs
- Results are returned or written to a table
Example
CREATE OR REPLACE PROCEDURE analyze_orders()
RETURNS TABLE(region VARCHAR, total_revenue FLOAT)
LANGUAGE PYTHON RUNTIME_VERSION = '3.11'
PACKAGES = ('snowflake-snowpark-python') HANDLER = 'run'
AS
$$
def run(session):
df = session.table("orders")
return df.group_by("region").agg({"amount": "sum"}).to_df("region", "total_revenue")
$$;
CALL analyze_orders();Key points
- Snowpark code runs on Snowflake's compute, not a separate cluster
- DataFrames are lazy; actions trigger SQL execution
- DataFrame syntax is familiar to pandas and PySpark users
- Python UDFs and procedures run in a secure sandbox on the warehouse
- Snowpark-optimized warehouses give more memory for heavy jobs
- Pick a currently supported Python runtime; 3.9 is deprecated