Lengthen Delta Sharing to Google Cloud Storage

This weblog article has been cross-posted from the Delta.io weblog.

We’re excited for the launch of Delta Sharing 0.4.0 for the open-source knowledge lake venture Delta Lake. The most recent launch introduces a number of key enhancements and bug fixes, together with the next options:

• Delta Sharing is now out there for Google Cloud Storage – Now you can share Delta Tables on the Google Cloud Platform (#81, #105)
• A brand new API for getting the metadata of a Delta Share – a brand new GetShare REST API has been added for querying a Share by its title (#95, #97)
• Delta Sharing Protocol and REST API enhancements – the Delta Sharing protocol has been prolonged to incorporate the Share Id and Desk Ids, as properly improved response codes and error codes (#85, #89, #93, #98)
• Customise a recipient sharing profile within the Apache Spark™ connector – a brand new Delta Sharing Profile Supplier has been added to the Spark connector to allow simpler entry of the sharing profile (#99, #107)

On this weblog submit, we are going to undergo every of the enhancements on this launch.

Delta Sharing on Google Cloud Storage

New to this launch, now you can share Delta Tables in Google Cloud Storage utilizing the reference implementation of a Delta Sharing Server.

Delta Sharing on Google Cloud Storage instance

Sharing Delta Tables on Google Cloud Storage is simpler than ever! For instance, to share a Delta Desk referred to as “time”, you possibly can merely replace the Delta Sharing server configuration with the placement of the Delta desk on Google Cloud Storage:

model: 1
shares:
- title: "vaccineshare"
 schemas:
 - title: "samplecoviddata"
   tables:
   - title: "time"
     location: "gs://deltasharingexample/COVID/Time"

Delta Sharing Server configuration file containing the placement to a Delta desk on Google Cloud Storage.
The Delta Sharing server will mechanically course of the info on Google Cloud Storage for a Delta Sharing question.
Authenticating with Google Cloud Storage
The Delta Sharing Server acts as a gatekeeper to the underlying knowledge in a Delta Share. When a recipient queries a Delta desk in a Delta Share, the Delta Sharing Server first checks the permissions to verify the info recipient has entry to knowledge. Subsequent, if entry is permitted, the Delta Sharing Server will have a look at the file objects that make up the Delta desk and well filter down the recordsdata if a predicate is included within the question, for instance. Lastly, the Delta Sharing Server will generate short-lived, pre-signed URLs that enable the info recipient to entry the recordsdata, or subset of recordsdata, from the Delta Sharing Consumer straight from cloud storage moderately than streaming the info by the Delta Sharing Server.

So as to generate the short-lived file URLs, the Delta Sharing Server makes use of a Service Account to learn Delta tables from Google Cloud Storage. To configure the Service Account credentials, you possibly can set the setting variable GOOGLE_APPLICATION_CREDENTIALS earlier than beginning the Delta Sharing Server.
# Delta Sharing Server Surroundings Variable

export GOOGLE_APPLICATION_CREDENTIALS="/config/keyfile.json"

New API for getting a Delta Share
Typically, it is likely to be useful for a recipient to test in the event that they nonetheless have entry to a Delta Share. This launch provides a brand new REST API, GetShare, in order that customers can shortly take a look at if a Delta Share has exceeded its expiration time.
For instance, to test in case you nonetheless have entry to a Delta Share you possibly can merely ship a GET request to the /shares/{share_name} endpoint on the sharing server:
import requests
import json

response = requests.get(
   "http://localhost:8080/delta-sharing/shares/airports",
   headers={
       "Authorization":"Bearer token"
   }
)
print(json.dumps(response.json(), indent=2))

Instance GET request despatched to the sharing server that allows recipients to test whether or not or not they nonetheless have entry to a Delta Share.
{
   "share": {
       "title": "airports"
   }
}

Instance response obtained from the GetShare REST API that’s new to the Delta Sharing 0.4.0 launch.
If the Delta Share has exceeded its expiration, the Sharing server will reply with a 403 HTTP error code.
Delta Sharing protocol enhancements
Included on this launch are improved error codes and error messages within the Delta Sharing protocol definition. For instance, if a Delta Share just isn’t situated on the Delta Sharing Server, an error code and error message containing the small print of the error is now included on this launch.
import requests
import json
 
response = requests.get(
   "http://localhost:8080/delta-sharing/shares/yellowcab",
   headers={
       "Authorization":"Bearer token"
   }
)
print(json.dumps(response.json(), indent=2))

Instance GET request for a Share that doesn’t exist on the Delta Sharing Server.
{
   "errorCode": "RESOURCE_DOES_NOT_EXIST",
   "message": "share 'yellowcab' not discovered"
}

Instance response containing an improved error code and particulars in regards to the error that’s new to the Delta Sharing 0.4.0 launch.
Moreover, this launch extends the Delta Sharing Protocol to reply with the distinctive Delta Share and Desk Ids. Distinctive Ids assist the info recipient disambiguate the title of datasets as time passes. That is particularly helpful when the info recipient is a big group and needs to use entry management on the shared dataset inside their group
Customizing a recipient Sharing profile
The Delta Sharing profile file is a JSON configuration file that incorporates the data for a recipient to entry shared knowledge on a Delta Sharing server. A brand new supplier has been added on this launch that allows simpler entry to the Delta Sharing profile for knowledge recipients.
/**
 * A supplier that gives a Delta Sharing profile for knowledge 
 * recipients to entry the shared knowledge. 
 */
trait DeltaSharingProfileProvider {
 def getProfile: DeltaSharingProfile
}

The Delta Sharing profile file is a JSON configuration file that incorporates the data for a recipient to entry shared knowledge on a Delta Sharing server.
What’s subsequent
We’re already gearing up for a lot of new options within the subsequent launch of Delta Sharing. You’ll be able to observe all of the upcoming releases and deliberate options in GitHub milestones.

Credit
We’d like to increase a particular thanks for the contributions to this launch to Denny Lee, Lin Zhou, Shixiong Zhu, William Chau, Xiaotong Solar, Kohei Toshimitsu.