S3
- Do not forget about bucket policy whe troubleshooting access
Find something by partial name
aws s3api list-objects-v2 --bucket bname --query "Contents[?contains(Key, 'searchstring')]"
Mount S3 as a file-system
- AWS S3 File Gateway - Expensive, I think this is an instance running NFS
https://github.com/s3fs-fuse/s3fs-fuse
- Why is it a bad idea? - Alternates listed
sudo apt-get install s3fs s3fs bucket-name /path/to/mountpoint -o passwd_file=/path/passwd-s3fs
S3 bucket policies
aws --profile myprod s3api get-bucket-policy --bucket bname |jq '.Policy | fromjson'
S3 list sorted
# All files sorted aws s3 ls --human-readable --recursive s3://bname/|sort|tee s3-files.sorted
Pulling all cloudWatch logs
for g in $(awslogs groups |grep interest); do echo '-------' echo $g awslogs get $g ALL -s6h done|tee 2019-04-02-my.logs
Uploading objects using presigned URLs
- Expiring early
- Don’t use role attached to instance. Create a special user with creds.
- Expiring early
Temp download link
# TTL is the number of seconds until the URL expires. # - 86400: 24 hours # - 604800: 7 days # - 2592000: 30 days TTL=604800 BUCKET=bucket-name OBJECT=object/path.tgz aws s3 presign s3://${BUCKET}/${OBJECT} --expires-in ${TTL}
https://github.com/s3gw-tech/s3gw
- Container able to run on a Kubernetes cluster, providing S3-compatible endpoints to applications.