1/3/2024 0 Comments Redshift create user example![]() ![]() Use the preprocessed data from Amazon S3 for any further training. Save the preprocessed data to /valohai/outputs so it gets saved to Amazon S3. It’s strongly recommended that you save the query result, or a preprocessed dataset, to /valohai/outputs/ to keep a snapshot of the exact query result before you run any training with that data.įor example, your pipeline could look like the one below:įetch data from Amazon Redshift and preprocess the data. Running a query today will return a certain query result but running the same query next week might return a different result. psql -h $redshift_host -d $dbname -U $user -p $port -A -f query.sql -F ',' -o /valohai/outputs/redshift_output.csv Maintaining reproducibilityĪs your data lives and changes in Amazon Redshift so will your query results. The code below will execute the query from query.sql (which is a part of the repository) and then output the results as a csv file to Valohai outputs. You can run psql directly in the mand your Docker image has it installed. Make sure you have psycopg2 in your Docker image, or install it with pip install psycopg2 in your mand before running your script. ![]() These environment variables will be now available for all executions that are run inside this project.īelow are two examples showing you how to access the environment variables during a Valohai execution. Which port are you connecting to? For example 5439.Ī username that can run operations in your Redshift database # Connect to Redshift cluster using AWS credentialsĪccess_key_id=credentials, # Fill in your details to these variables Response = requests.get(f' # Parse the JSON results # Fetch credentials from the machines ValohaiWorkerRole Here’s a simple example: import redshift_connector You can use for example redshift_connector connect to Redshift from your execution. You can create multiple roles and have Valohai environments that are connected to different roles.įor example, creating a ValohaiWorkerRole-Redshift that will be used only by certain Valohai environments, and those can be restricted to only certain teams in your Valohai organization. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |