Upload file result.csv into Azure Data Lake (in mystorageaccount/mycontainer).

Databricks commands:

# ADLS Key
spark.conf.set(
  'fs.azure.account.key.mystorageaccount.dfs.core.windows.net'
  , 'my_key'
)

spark.conf.set() define the access key for the connection to Data Lake. The access key can be found in Azure Portal.

# List folder in ADLS
dbutils.fs.ls('abfss://mycontainer@mystorageaccount.dfs.core.windows.net/Test_CSV')

To list a folder in ADLS, use dbutils.fs.ls().

# Read file from ADLS
source_file = 'abfss://mycontainer1@mystorageaccount.dfs.core.windows.net/result.csv'

df = spark.read\
  .format('csv')\
  .option('inferSchema', True)\
  .option('header', True)\
  .option('delimiter', ',')\
  .load(source_file)

display(df)

To read a file in ADLS, use spark.read(). The result is inserted in a DataFrame (df). Next it can be manipulated in Databricks.

To browse the DataFrame – display(df).

# Show the schema
df.printSchema()

To show the schema of the DataFrame –  df.printSchema().

# Create temp view from the DataFrame
df.createOrReplaceTempView('result_temp_view')

Create a temporary view in Databricks that will allow the manipulation of the data.

%sql
-- Manipulate the DataFrame with SQL
CREATE OR REPLACE TEMPORARY VIEW result_temp_view_1 AS
SELECT
  *
  , CASE
    WHEN score > 200 THEN 'Good'
    ELSE 'Bad'
  END AS derived_column
FROM result_temp_view
ORDER BY score

Use SQL to create a temporary view and to add derived column.

# Create DataFrame from temporary view
df1 = spark.sql('SELECT * FROM result_temp_view_1')

Create DataFrame from temporary view.

# Save the DataFrame in file on ADLS
destination_parh = 'abfss://mycontainer@mystorageaccount.dfs.core.windows.net'

# Parquet (insert the result from the temporary view)
df1.write.format('parquet')\
  .mode('overwrite')\
  .save(destination_parh + '/Databricks_Parquet')

# JSON
df.write.format('json')\
  .mode('overwrite')\
  .save(destination_parh + '/Databricks_JSON')

# CSV
df.write.format('csv')\
  .mode('overwrite')\
  .option('header', True)\
  .option('sep', ',')\
  .save(destination_parh + '/Databricks_CSV')

Save DataFrame in Parquet, JSON or CSV file in ADLS. The output is saved in Delta Lake – an open-source storage layer that brings ACID (atomicity, consistency, isolation, and durability) transactions to Apache Spark and big data workloads.

The files in Delta Lake are partitioned and they do not have friendly names:

# Read Parquet Delta Lake
df_parquet = spark.read\
  .parquet(destination_parh + '/Databricks_Parquet')

display(df_parquet)

Read Parquet Delta Lake:

# Read JSON Delta Lake
df_json = spark.read\
  .json(destination_parh + '/Databricks_JSON')

display(df_json)

Read JSON Delta Lake.

# Read CSV Delta Lake
df_csv = spark.read\
  .option('header', True)\
  .csv(destination_parh + '/Databricks_CSV')

display(df_csv)

Read CSV Delta Lake.

Keep it simple :-)