Notebooks & Spark

Launching a notebook

1. Applications → Notebook → Open (or notebook.<domain>). You sign in through SSO.

2. Pick a server size and click Start. Spawning takes up to a few minutes.

   Size	Limit	Guaranteed
   Small	1 CPU / 2 GiB	0.25 CPU / 512 MiB
   Medium (default)	2 CPU / 4 GiB	0.5 CPU / 1 GiB
   Large	4 CPU / 8 GiB	1 CPU / 2 GiB

   The kernel's memory is for Python only. Spark work runs on the shared Spark server, not in your notebook. Start with Medium, and use Large only for heavy pandas work.

3. Your home folder (/home/jovyan/work, 10 GiB per user) is persistent. Anything else in the container is not.

4. Idle servers stop after 30 minutes (checked every 5 minutes). Save often. Long computations should be dbt models or jobs, not notebook cells.

How Spark works here

• Each workspace has one shared Spark 4 Connect server. Your notebook is a thin client. Data never flows through your kernel unless you collect it.
• Executors start on demand and are released after about 2 minutes idle. The first query after a pause takes about 20–40 s. After that, queries are fast.
• Executor memory and cores are set on the server. You can tune session settings such as shuffle partitions, but not executor size.

Using Spark in a notebook

Helpers are pre-loaded in every kernel. On start you'll see [datanature] tenant=… warehouse=… ready.

sql("SHOW NAMESPACES IN lakehouse").show()

df = table("bronze_crm.customers")          # = lakehouse.bronze_crm.customers
df.groupBy("country").count().show()

s = spark()                                 # the full SparkSession
s.conf.set("spark.sql.shuffle.partitions", "400")   # usually unnecessary: AQE is on

# write an Iceberg table
(df.filter("status = 'ACTIVE'")
   .writeTo("lakehouse.sandbox_alice.active_customers")
   .using("iceberg")
   .createOrReplace())

# append
new_rows.writeTo("lakehouse.sandbox_alice.active_customers").append()

from datanature import status               # connection diagnostics (import explicitly)
status()

Notes:

• The catalog is always lakehouse. table() needs at least schema.table.
• Tables you write are immediately queryable from the Workbench, BI and dbt.
• A generic client (any workspace pod): SparkSession.builder.remote(os.environ["SPARK_REMOTE"]).getOrCreate().
• SPARK_REMOTE is not set means you're outside a workspace notebook or dbt shell.

Working with data sizes

Do	Don't
Aggregate or filter in Spark, then .toPandas() the small result	.toPandas() a multi-GB table. The kernel has only 2–8 GB.
Write big outputs as Iceberg tables	Collect big results into Python lists
Use a personal sandbox schema (sandbox_<you>) for experiments	Write into bronze, silver or gold from notebooks; those belong to pipelines and dbt
Move a notebook that has become a pipeline into a dbt model + job	Schedule production logic as a notebook

ML, vectors and caches from notebooks

These tools have no web UI and are reachable only from inside the platform (e.g. from a notebook):

import mlflow
mlflow.set_tracking_uri("http://mlflow.mlops.svc.cluster.local:5000")   # artifacts go via the server

from pymilvus import MilvusClient
milvus = MilvusClient(uri="http://milvus.vector.svc.cluster.local:19530")

import redis
r = redis.Redis(host="redis-stack.cache.svc.cluster.local", port=6379, password="<ask admin>")

See BI, Voir AI & ML Tools for limits.

Troubleshooting notebooks

Symptom	Cause	Do
"Spawn failed:" with an empty reason after about 5 min, often after maintenance	The hub lost track of servers	Stop retrying after 2 attempts. The hub shuts itself down after 5 consecutive failures. Contact your admin (restarting the hub is safe).
First query slow (20–40 s)	Executors starting from zero	Normal
NotAuthorized, S3V4RestSignerClient.sign, Failed to authenticate	The shared Spark server's catalog session went stale. It affects everyone, so restarting your kernel alone won't fix it.	Copy the full stack trace and contact your admin (Spark restart)
INVALID_HANDLE / session errors after the above is fixed	Spark server was restarted	Kernel → Restart
ValueError … needs at least namespace.table / table not found	Unqualified name	Use schema.table or lakehouse.schema.table
Kernel dies / out of memory	Collected too much data into pandas	Aggregate in Spark first