Implement uv_fs_mkdtemp #1368

Hinidu · 2014-07-13T20:07:07Z

I implemented uv_fs_mkdtemp for unices and created a test for that. Could someone confirm that I am on the right way or provide some suggestions before I start implementing uv_fs_mkdtemp for Windows? Specifically I'm in doubt whether uv_fs_t->ptr is the right place for storing the name of temp directory which was created by underlying mkdtemp.

For Windows I want to use glibc's implementation which uses __gen_tempname.

Should close #1315

saghul · 2014-07-13T22:02:14Z

src/unix/fs.c

+                  const char* template,
+                  uv_fs_cb cb) {
+  INIT(MKDTEMP);
+  req->ptr = strdup(template);


I'd use req->path.

Hinidu · 2014-07-14T07:49:58Z

@saghul thank you for suggestions! Does it seem good now?

saghul · 2014-07-14T07:52:05Z

src/unix/fs.c

@@ -991,6 +997,16 @@ int uv_fs_mkdir(uv_loop_t* loop,
 }


+int uv_fs_mkdtemp(uv_loop_t* loop,
+                  uv_fs_t* req,
+                  const char* path,


Call this 'template'.

Hinidu · 2014-07-14T08:12:51Z

@saghul fixed last two issues.

saghul · 2014-07-14T08:17:55Z

test/test-fs.c

@@ -787,6 +803,12 @@ TEST_IMPL(fs_async_dir) {
  uv_run(loop, UV_RUN_DEFAULT);
  ASSERT(mkdir_cb_count == 1);

+  r = uv_fs_mkdtemp(loop, &mkdtemp_req, "test_dir_XXXXXX", mkdtemp_cb);


Better add a new test, even if you need to copy some bolierplate. Also, if you create the file inside a temp directory you can just remove the entire directory, instead of saving the filename and removing that one then.

Ok, I'll add new test. I didn't create the file inside a temp directory yet, I'll add code for that in test too.
Btw how can I remove entire directory? Does uv_fs_rmdir remove directory recursively?

Btw how can I remove entire directory? Does uv_fs_rmdir remove directory recursively?

I don't recall from the top of my head, but I think some test already does it...

Hinidu · 2014-07-18T13:33:36Z

@saghul I've pushed the new version of test implementation. Do you (or anyone else) have other suggestions?

saghul · 2014-07-18T13:42:28Z

test/test-fs.c

+  ASSERT(req == &mkdtemp_req);
+  check_mkdtemp_result(req);
+  mkdtemp_cb_count++;
+  strcpy(buf, req->path);


Better remove the file here, instead of copying the path to buf.

I did this to compare results of two uv_fs_mkdtemp calls. But I can also implement it by adding mkdtemp_req2 and by calling uv_fs_req_cleanup in test body instead of mkdtemp_cb. Is this more appropriate way for you?

Sounds good!

Hinidu · 2014-07-19T06:55:07Z

@saghul thank you again for review! I've implemented your suggestions.

txdv · 2014-07-19T07:48:45Z

src/unix/fs.c

+                  const char* template,
+                  uv_fs_cb cb) {
+  INIT(MKDTEMP);
+  req->path = strdup(template);


Are you freeing this anywhere?

It's freed on request cleanup internally.
On Jul 19, 2014 9:48 AM, "Andrius Bentkus" notifications@github.com wrote:

In src/unix/fs.c:

@@ -991,6 +997,18 @@ int uv_fs_mkdir(uv_loop_t* loop,
}

+int uv_fs_mkdtemp(uv_loop_t* loop,

uv_fs_t\* req,

const char\* template,

uv_fs_cb cb) {

INIT(MKDTEMP);

req->path = strdup(template);

Are you freeing this anywhere?

—
Reply to this email directly or view it on GitHub
https://github.com/joyent/libuv/pull/1368/files#r15142598.

https://github.com/joyent/libuv/blob/master/src/unix/fs.c#L1154

https://github.com/Hinidu/libuv/blob/mkdtemp/src/unix/fs.c#L1173

everywhere we use free(var), but here we are using free((void *) var);

The user has to actually clean up after the operation is finished by calling uv_fs_req_cleanup.

Of course the user has to call that function, that's how it works. Not sure
what your concern is.
On Jul 19, 2014 9:58 AM, "Andrius Bentkus" notifications@github.com wrote:

In src/unix/fs.c:

@@ -991,6 +997,18 @@ int uv_fs_mkdir(uv_loop_t* loop,
}

+int uv_fs_mkdtemp(uv_loop_t* loop,

uv_fs_t\* req,

const char\* template,

uv_fs_cb cb) {

INIT(MKDTEMP);

req->path = strdup(template);

https://github.com/Hinidu/libuv/blob/mkdtemp/src/unix/fs.c#L1173

everywhere we use free(var), but here we are using free((void *) var);

The user has to actually clean up after the operation is finished by
calling uv_fs_req_cleanup.

—
Reply to this email directly or view it on GitHub
https://github.com/joyent/libuv/pull/1368/files#r15142632.

@txdv sorry, I don't understand what you want to say in your last comment, but I want to point you to https://github.com/Hinidu/libuv/blob/mkdtemp/src/unix/fs.c#L91
This PATH as you can see is used in many functions and it does the same thing as my code here. Actually I written it explicit by @saghul's suggestion because PATH works with path argument but in case of uv_fs_mkdtemp argument is template.

@Hinidu nevermind, I was wrong

@saghul you said it would be done internally, but it is actually handled by the user. it is a small technical difference

Well, what I meant is that you don't explicitly need to free it. It
happens as part of the cleanup process. It's an internal detail the user
doesn't really need to know about.
On Jul 19, 2014 10:27 AM, "Andrius Bentkus" notifications@github.com
wrote:

In src/unix/fs.c:

@@ -991,6 +997,18 @@ int uv_fs_mkdir(uv_loop_t* loop,
}

+int uv_fs_mkdtemp(uv_loop_t* loop,

uv_fs_t\* req,

const char\* template,

uv_fs_cb cb) {

INIT(MKDTEMP);

req->path = strdup(template);

@Hinidu https://github.com/Hinidu nevermind

@saghul https://github.com/saghul you said it would be done internally,
but it is actually handled by the user. it is a small technical difference

—
Reply to this email directly or view it on GitHub
https://github.com/joyent/libuv/pull/1368/files#r15142718.

Yes, I thought I saw a bug so I checked it out. Sometimes I am wrong.

No worries! It's great to have an extra pair of eyes on the watch, thanks!
:-)
On Jul 19, 2014 10:37 AM, "Andrius Bentkus" notifications@github.com
wrote:

In src/unix/fs.c:

@@ -991,6 +997,18 @@ int uv_fs_mkdir(uv_loop_t* loop,
}

+int uv_fs_mkdtemp(uv_loop_t* loop,

uv_fs_t\* req,

const char\* template,

uv_fs_cb cb) {

INIT(MKDTEMP);

req->path = strdup(template);

Yes, I thought I saw a bug so I checked it out. Sometimes I am wrong.

—
Reply to this email directly or view it on GitHub
https://github.com/joyent/libuv/pull/1368/files#r15142758.

saghul · 2014-07-22T21:05:39Z

@Hinidu ping, do you think you'll have time to complete the Windows part soon?

Hinidu · 2014-07-22T21:20:08Z

@saghul actually I thought to ask tomorrow whether you have any other suggestions or I can proceed with Windows part :-) I hope tomorrow I will have time to implement uv_fs_mkdtemp for Windows. I think that it would be the most complex part of this PR.
Btw if you don't have objections I will squash last two commits for the unittest into one.

saghul · 2014-07-22T21:27:22Z

Fantastic news! :-) Don't worry about squashing, it can be done at the end, once we have the code ready.

Hinidu · 2014-07-23T07:40:56Z

@saghul I've pushed implementation for Windows. Make it work was simpler than I thought in the beginning :-) Now it passes my test locally.
Actually I ported glibc's implementation which uses __gen_tempname.
I have some questions which I'll post under particular lines shortly.

Hinidu · 2014-07-23T07:43:27Z

src/win/fs.c

@@ -721,6 +721,80 @@ void fs__mkdir(uv_fs_t* req) {
 }


+void fs__mkdtemp(uv_fs_t* req) {
+  static const WCHAR letters[] =
+    L"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";


Does it make sense to use both upper and lower case? AFAIK most Windows file systems don't distinguish them in file names.

IIRC, NTFS can, so yes.
On Jul 23, 2014 9:43 AM, "Pavel Platto" notifications@github.com wrote:

In src/win/fs.c:

@@ -721,6 +721,80 @@ void fs__mkdir(uv_fs_t* req) {
}

+void fs__mkdtemp(uv_fs_t* req) {

static const WCHAR letters[] =

L"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";

Does it make sense to use both upper and lower case? AFAIK most Windows
file systems don't distinguish them in file names.

—
Reply to this email directly or view it on GitHub
https://github.com/joyent/libuv/pull/1368/files#r15274807.

Hinidu · 2014-07-23T08:50:11Z

@saghul I've pushed your suggestions.

saghul · 2014-07-23T08:52:44Z

Nice! I'll take a deeper look tonight, thanks!

Hinidu · 2014-07-23T08:56:59Z

Thank you, I'll wait final review!

saghul · 2014-07-23T21:50:26Z

src/win/fs.c

+  /* Get some random data. */
+  value += uv_hrtime() ^ _getpid();
+
+  /* Magic number 7777 was borrowed from the glibc's implementation */


Please move the comment to outside the function, just say it's borrowed from glibc.

/* Some parts of the implementation was borrowed from glibc. */

Does this comment look good to you?

saghul · 2014-07-23T22:22:09Z

Other than clarifying to me the wide char thing, LGTM. Test pass: http://jenkins.nodejs.org/job/libuv-pullrequest/631/ Can you PTAL, @indutny?

Hinidu · 2014-07-24T06:32:01Z

@saghul fixed comment, squashed.

saghul · 2014-07-24T07:45:43Z

src/win/fs.c

+  WCHAR* template_part;
+  static uint64_t value;
+  unsigned int count;
+  int fd = -1;


No need to initialize this, right?

Hinidu · 2014-07-24T08:18:42Z

Should I do git commit --amend && git push --force for those small changes or should I add new commit for them to squash it after final review?

saghul · 2014-07-24T08:27:02Z

Just force push, thanks!
On Jul 24, 2014 10:18 AM, "Pavel Platto" notifications@github.com wrote:

Should I do git commit --amend && git push --force for those small
changes or should I add new commit for them to squash it after final review?

—
Reply to this email directly or view it on GitHub
#1368 (comment).

Hinidu · 2014-07-24T08:30:03Z

@saghul Done!

saghul · 2014-07-30T08:56:08Z

@indutny Ping.

indutny · 2014-07-30T09:35:04Z

src/unix/fs.c

+  INIT(MKDTEMP);
+  req->path = strdup(template);
+  if (req->path == NULL)
+    return -ENOMEM;


What about ENOTDIR ?

nvm, it is returned for other functions of this family.

This is just checking the result after strdup.

indutny · 2014-07-30T09:37:32Z

LGTM, if it works

saghul · 2014-07-30T09:41:31Z

Tests pass, will land tonight then. Thanks for the review Fedor!

Hinidu · 2014-07-30T09:47:46Z

LGTM, if it works

Thank you for review!

Tests pass, will land tonight then.

Great! This code will go to 0.11.27? Is there an approximate release date of this version?

saghul · 2014-07-30T09:52:38Z

Great! This code will go to 0.11.27? Is there an approximate release date of this version?

Not sure if we'll release any libuv 0.11.x before 0.12. About the release date: it will be ready when it's ready :-)

Hinidu · 2014-07-30T09:58:16Z

Not sure if we'll release any libuv 0.11.x before 0.12. About the release date: it will be ready when it's ready :-)

Ok, then I'll try to keep an eye on this so I hope I don't miss next release :-)

saghul · 2014-07-31T07:57:19Z

Landed in master, thanks for working on this @Hinidu!

Hinidu · 2014-07-31T07:59:44Z

Landed in master, thanks for working on this @Hinidu!

I'm very glad to hear it! Thank you and all other collaborators for your great library!

Adds new `uv_fs_mkdtemp` function which we need for cross-platform temp directory creation. ref: - neovim#812 - joyent/libuv#1368 - https://github.com/joyent/libuv/releases

sam-github · 2014-08-06T22:06:11Z

src/win/fs.c

@@ -721,6 +721,78 @@ void fs__mkdir(uv_fs_t* req) {
 }


+/* Some parts of the implementation were borrowed from glibc. */


Is incorporating LGPL source into an MIT licensed project legal?

It's clearly recognizable as glibc's __gen_tempname() from sysdeps/posix/tempname.c, parts have been copied verbatim. This seems like murky legal water.

Ouch. My bad since I did the reviewing :-( I'll look for an alternative with appropriate licensing. If anyone gets there beforehand, please open an issue / PR.

I apologize. It was my mistake. I hadn't much experience with copyright law.
I found slightly different OpenBSD version here http://openbsd.cs.toronto.edu/cgi-bin/cvsweb/src/lib/libc/stdio/mktemp.c It seems like their license permits copying and modification.
I can try change current implementation according to this version.
Sorry again, I hope I didn't create a lot of problems.

saghul · 2014-08-07T07:16:55Z

I think we can go with the MIT licensed musl libc's implementation: https://github.com/runtimejs/musl-libc/blob/master/src/temp/mkdtemp.c and https://github.com/runtimejs/musl-libc/blob/master/src/temp/__randname.c.

Same license, so there should be no problem.

Hinidu · 2014-08-07T07:22:02Z

@saghul I've pushed #1402 right now. But if musl's version is more desirable than I can use it.

saghul · 2014-08-07T07:26:17Z

Oh, it's ok, will review now.

saghul reviewed Jul 13, 2014
View reviewed changes

src/unix/fs.c

const char* template,

uv_fs_cb cb) {

INIT(MKDTEMP);

req->ptr = strdup(template);

Copy link

Contributor

saghul Jul 13, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd use req->path.

saghul reviewed Jul 14, 2014
View reviewed changes

This was referenced Jul 14, 2014

[RDY] Implement mkdtemp for Windows and refactor vim_tempname etc. neovim/neovim#850

Closed

Create portable os_ function to get temporary filenames neovim/neovim#812

Closed

saghul reviewed Jul 18, 2014
View reviewed changes

txdv reviewed Jul 19, 2014
View reviewed changes

saghul added this to the v0.12 milestone Jul 19, 2014

Hinidu reviewed Jul 23, 2014
View reviewed changes

saghul reviewed Jul 23, 2014
View reviewed changes

saghul reviewed Jul 24, 2014
View reviewed changes

Implement uv_fs_mkdtemp

a1dea98

saghul mentioned this pull request Jul 30, 2014

Implement cross-platform mkdtemp #1315

Closed

indutny reviewed Jul 30, 2014
View reviewed changes

saghul closed this Jul 31, 2014

Hinidu mentioned this pull request Aug 5, 2014

[RDY] Implement os_mkdtemp on top of uv_fs_mkdtemp neovim/neovim#1034

Closed

sam-github reviewed Aug 6, 2014
View reviewed changes

saghul mentioned this pull request Aug 7, 2014

fs: replace mkdtemp implementation #1403

Closed

ralt mentioned this pull request Feb 19, 2016

Add the fs.mkdtemp() function. nodejs/node#5333

Closed

		@@ -721,6 +721,78 @@ void fs__mkdir(uv_fs_t* req) {
		}


		/* Some parts of the implementation were borrowed from glibc. */

Implement uv_fs_mkdtemp #1368

Implement uv_fs_mkdtemp #1368

Conversation

Hinidu commented Jul 13, 2014

Choose a reason for hiding this comment

Hinidu commented Jul 14, 2014

Choose a reason for hiding this comment

Hinidu commented Jul 14, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hinidu commented Jul 18, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hinidu commented Jul 19, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saghul commented Jul 22, 2014

Hinidu commented Jul 22, 2014

saghul commented Jul 22, 2014

Hinidu commented Jul 23, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hinidu commented Jul 23, 2014

saghul commented Jul 23, 2014

Hinidu commented Jul 23, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saghul commented Jul 23, 2014

Hinidu commented Jul 24, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hinidu commented Jul 24, 2014

saghul commented Jul 24, 2014

Hinidu commented Jul 24, 2014

saghul commented Jul 30, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indutny commented Jul 30, 2014

saghul commented Jul 30, 2014

Hinidu commented Jul 30, 2014

saghul commented Jul 30, 2014

Hinidu commented Jul 30, 2014

saghul commented Jul 31, 2014

Hinidu commented Jul 31, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saghul commented Aug 7, 2014

Hinidu commented Aug 7, 2014

saghul commented Aug 7, 2014