feature request: aws s3 cat command

[adamantnz]

Hi team. It'd be great to have an aws s3 cat command to preview text files via the aws cli. I often have to COPY data into Redshift from S3 and having the ability to see the first few rows would be super handy for seeing whether there is a header, what data types I would need in my schema, what delimiter has been used etc.

Thanks!

[joguSD]

Marking as a feature request.

This is more or less an alias of something like the following:

aws s3 cp s3://bucket/key -

Which means copy the object to standard out.

[mr-karan]

I'd like to work on this

[JordonPhillips]

@joguSD technically that isn't cat since you wouldn't be able to concatenate multiple files

[thinkerbot]

Huge fan of this. I think it would be nice if this allowed fetch based on both s3 and https urls.

[pgollucci]

you can do it with aws s3 cp make an alias in ~/.aws/cli/alias

…

On Tue, Dec 12, 2017 at 8:58 AM, Simon Chiang ***@***.***> wrote: Huge fan of this. I think it would be nice if this allowed fetch based on both s3 and https urls. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2982 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACF97vEtr71Q9trn2IDazr6KCU-r0Feks5s_rCZgaJpZM4Qogk1> .

-- --------------------------------------------------------------------------------- 4096R/D21D2752 <http://pgp.mit.edu/pks/lookup?op=get&search=0xF699A450D21D2752> ECDF B597 B54B 7F92 753E E0EA F699 A450 D21D 2752 Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Cloud Technology, Capital One What doesn't kill us can only make us stronger; Except it almost kills you.

[dannypaz]

First, this would be a great feature to have (as im currently looking into solutions for cat functionality)

In the meantime, AWS has posted an article on doing concatenation of objects w/ the ruby sdk

This is specifically using multipart_upload and copy_part.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[chrispruitt]

any update on this?

[AlJohri]

This would be a great addition. Going back and forth between ls and "cp with a dash at the end" is fairly tedious.

[kelleysy]

this would be really awesome 🙏

[bonespiked]

doing an effective +1 - I would also suggest adding in a 'head' option:
aws s3 head s3://my-bucket/big-file

[lorenzfischer]

You can do a cat using the - option with the cp command (see this thread on Stackoverflow):

aws s3 cp s3://path_to_your_file.gz - | gunzip | less -S

This way, you can also head things:

aws s3 cp s3://path_to_your_file.gz - | gunzip | head -n 10 | less -S

Disclaimer: I'm not sure about how this works in the background (i.e. what exactly it loads where), but less opens fairly quickly on a large file, so it seems to do streaming. I'm also not sure how save it is to suppress the error message about the 'broken pipe' when the stream is stopped (for example with head).

Hope this is helpful.

[AAAlexandre]

Hello.
Sorry for hijacking this feature request, but since the support redirected me to here, I'll try to make sure my request is also visible.

aws s3 cat would indeed be very helpful, especially for catenating multiple files.
The current alternative, <list xargs -I% aws s3 cp % - is excruciatingly slow (less than 50 files per minute) and the reason is only because of the authentication procedures which take up over 90% of the time.
The alternative using hdfs and s3a:// is way more effective: <list xargs hdfs dfs -cat yields almost 1000 files per minute.
aws s3 cp --recursive is also blazingly fast (but requires sufficient disk space): 6500 files per minute.

It would also be enough to enable the --recursive option on cp to -:
aws s3 cp --recursive s3://.../ --exclude '*' --include '...' -

==> Please consider this feature seriously.

[EugeneChung]

When would this happen to be added? I think it's not that difficult.