To read CSV data, you can make use of Custom R Script, after installing aws.s3 package from CRAN.
Here is an example of such a script.
library(aws.s3)
# Set key, secret
AWS_KEY<- "xxxxxx"
AWS_SECRET <-"xxxxxx"
# Read CSV from S3
read.csv(text = rawToChar(aws.s3::get_object(object = paste0("s3://my_bucket/my_data.csv"), key=AWS_KEY, secret=AWS_SECRET)))