Data Lakehouse with PySpark — Setup Delta Lake Warehouse on S3 and Boto3 with AWS

Next, as part of the series Data Lakehouse with PySpark, we need to setup boto3 and Delta Lake to communicate with AWS S3. This will help us to create our default warehouse location for Delta Lake on AWS S3. We will also setup the metastore location for Delta Lake.

Representation Image
To start, we need the AWS credentials Access Key and Secret Access Key. Checkout — to know more. In case of any issues, please follow the YouTube video at the end.

Connect AWS from boto3:

Once we have the AWS Access Key and Secret Access Key, create a new folder .aws and file credentials in the user’s root directory. Add the following lines replacing the Access Key, Secret Key with profile as default and save the file.

Credential file
aws_access_key_id=<Your AWS Access Key>
aws_secret_access_key=<Your AWS Secret Key>

And we are done, now boto3 can easily use the credentials from the default profile to connect with AWS.

Connect AWS from Delta Lake:

To connect Delta Lake with AWS S3 and create the default warehouse location on AWS S3. Add the following lines in the bottom on spark-defualts.conf file.

spark.sql.warehouse.dir s3a://easewithdata/dw-with-pyspark/warehouse
spark.driver.extraJavaOptions -Dderby.system.home=/home/jupyter/ease-with-data/dw-with-pyspark/derby
Spark Defaults Conf Setup
Please change parameter as per your location on S3. This setup is done as per the session of Data Lakehouse on YouTube —

We can also define the location of the metastore for Delta Lake using the hive-site.xml file.

<description>JDBC connect string for a JDBC metastore</description>
<description>Driver class name for a JDBC metastore</description>
<description>location of default database for the warehouse</description>
Change Metastore Location

Save the files and we would now be able to easily create Delta table with default warehouse location on S3.

DB and tables creates in S3

Github location for conf files —

Still struggling, checkout the following YouTube video

Make sure to Like and Subscribe.

Follow us on YouTube:

If you are new to Data Lakehouse checkout —

Buy me a Coffee

If you like my content and wish to buy me a COFFEE. Click the link below or Scan the QR.
Buy Subham a Coffee
*All Payments are secured through Stripe.

Scan the QR to Pay Securely

About the Author

Subham is working as Senior Data Engineer at a Data Analytics and Artificial Intelligence multinational organization.
Checkout portfolio: Subham Khandelwal