io/ioutil: predictable suffix for TempFile

[gopherbot]

Right now ioutil.TempFile creates files with a given prefix as the basename. This is
fine for the usual /tmp usage, where all the files in that directory are temporary files.

However, for e.g. replacing a state file ~/.foo.json with a new one, what I'd want is
writing to e.g. ~/.foo.<RANDOMNESS>.tmp and renaming that over ~/.foo.json. This
way I can make e.g. backup and editor software ignore *.tmp files, and even clean old
leftover temp files automatically.

[adg]

Comment 1:

Labels changed: added priority-someday, packagechange, removed priority-triage, go1.1maybe.

Status changed to Accepted.

[rsc]

Comment 2:

You could also use .tmp.foo. as your prefix.

[rsc]

Comment 3:

Labels changed: added repo-main.

[rsc]

Comment 4:

Adding Release=None to all Priority=Someday bugs.

Labels changed: added release-none.

[odeke-em]

I have run into this

See cmd/pprof/internal/tempfile has a nice function https://golang.org/pkg/cmd/pprof/internal/tempfile/#New

tempfile.New(dir, prefix, suffix string) (*os.File, error)

although imports of internal packages aren't allowed in >=Go1.6.

Yet

ioutil.TempFile(dir, prefix string)

and adds a suffix to the end of the path, and this messes up the intended suffix

My use case is for programs that need the suffix to try to select the opener by using the suffix/extension e.g

$ file pliesHello
pliesHello: ISO Media, MPEG v4 system, version 2
$ open pliesHello

cmd/pprof/internal/tempfile IMO provides a better API, since it even allows for DeferPath(path) deletion and no need to Close() then os.RemoveAll()
Any thoughts?

[davecheney]

@odeke-em maybe I don't understand the issue, but can't you just os.Rename the temp file once it's been created ?

[odeke-em]

Yap, same train of thought. For now that's exactly what am doing.

However, notice the number of steps that it takes:

• ioutil.TempFile(entryDir, prefix).
• defer tempFile.Close().
• defer os.RemoveAll(path.Join(entryDir, tempFile.Name()))
• and all the error checking.

os.Rename suffers from a few problems if the /tmp dir is not on the same device
unless we do byte by byte copying, please see rakyll/statik#5.
It also isn't atomic.

Now let's compare that to using tempfile.New, and tempfile.DeferDelete and that's simpler.
I am all for us keeping things as they were if it keeps the code and language simpler and if the implementation is tricky, then I can implement some package or ways around it.

[hgl]

Some tools (e.g., vips) rely on the extension to determine how to process the file. When exec such commands and feeding them temp files, being able to specify suffix is very important.

Right now I have to copy https://golang.org/pkg/cmd/pprof/internal/tempfile from an early go to a new package to achieve that.

[JensRantil]

As a workaround, we just open sourced a tiny library which supports suffix, too: https://github.com/tink-ab/tempfile

[TomOnTime]

This is even more important for Windows systems where extensions mean something. For example, in one project we needed to have a ".bat" extension.

[TomOnTime]

This seems like a good one for community day. Can we add a "help wanted" label?

[TomOnTime]

I think I'm going to volunteer for this.

I'm new to contributing to this project, so hand holding is appreciated.

According to Contribution Guide, I should discuss the design a bit. Here is my design proposal:

• ioutil.TempFileExt() will be like TempFile() but take an extra parameter, the extension.
• The caller will have to include the entire extension, including the dot: .bat not bat
• refactor TempFile() to call TempFileExt(a, b, "")
• rename helper function nextSuffix() to nextRandText() to be more accurate.
• add unit tests

[hgl]

Thanks for your work @TomOnTime.

I'm not sure if I'm overengineering this, but I don't think it needs to have anything to do with extensions.

What if we use a more generic solution:

ioutil.TempFileTemplate(dir, "{{x}}file.bat")

Where {{x}} (mimicking the syntax introduced in text/template) is the random number that will be inserted by the function. The syntax and other restrictions (e.g., only one {{x}} is allowed, otherwise panic) are debatable.

[adamdecaf]

The caller will have to include the entire extension, including the dot: .bat not bat

If the caller does not provide an extension will bat just be appended, or are you saying that should be an error?

I'll vote for ioutil.TempFileExt(). path and path/filepath use *Ext() already. I think making it a template opens up too many error conditions.

[TomOnTime]

The caller will have to include the entire extension, including the dot: .bat not bat

If the caller does not provide an extension will bat just be appended, or are you saying that should be an error?

I just mean that the the filename is made of prefix + random + extension, therefore callers will usually call TempFile() with a suffix that includes the "." if they want a ".".

Callers might assume the formula is prefix + random + "." + extension and they would be wrong.

TempFile(a, b) is simply a call to TempFileExt(a, b, "")

I think TempFileTemplate is a bit overly complex for the needs here.

[TomOnTime]

FYI:

Technically the 3rd parameter is a suffix, not an extension. A better name is TempFileSuffix(p,n,s)

[TomOnTime]

Makes sense. I'm new here so... what's next?

[dsnet]

Makes sense. I'm new here so... what's next?

Welcome and thanks for interest in contributing :)

The formal process is here and is intended to ensure that new features are bring sufficient benefit to warrant expanding the API, and don't bring along with it major detriments.

Periodically (I think weekly?), a group of proposal reviewers scrub through issues in the proposal milestone and determine whether to approve/disapprove the suggested API change. However, there is a large backlog of proposals, so it may take time for them to get around to this one.

[rsc]

There's ioutil.TempDir already, that's one option if you want control over file suffixes.

We don't yet know how we'd want to add support for suffixes, or if we really do. One option instead of TempFile2 is to define that the current prefix is a pattern so that if there's a * that's where the randomness goes.

ioutil.TempFile("", "foo*.bat")

etc.

[ianlancetaylor]

Does @rsc's suggestion above address the use case here?

[TomOnTime]

Does @rsc's suggestion above address the use case here?

The use of "*" as a marker for where to insert the random bits? Yes, that would work.

[Moataz-E]

This would be a very welcome change guys, a huge functionality improvement.

[rsc]

OK, let's do what I wrote above: if "*" is in the string, then replace it with the random part. If there are multiple stars, use the first one (we have to pick something).

[TomOnTime]

I volunteer to write this. Seems like a good starter-project.

[tandr]

Personally, I like predictability and "following the spirit of existing API" of an original TempFileSuffix proposal more.

IMHO there are too many problems with *

• If only first * is going to be replaced, how long should that part be, "enough to be unique" or some predefined minimum?
• What if there are multiple *, how long is long enough?
• I want a file with * in it (yes, exceptionally rare case), what do I do? Escape it?

[TomOnTime]

The new implementation replaces the first *. If you want an actual * in the file name, just include it after the first *.

[tandr]

@TomOnTime that way "Temp*.RANDOMPART.json" will still be impossible

[TomOnTime]

@TomOnTime that way "Temp*.RANDOMPART.json" will still be impossible

@tandr Agreed.We're making it difficult to do pathologically bad things.

[gopherbot]

Change https://golang.org/cl/105675 mentions this issue: io/ioutil: TempFile should permit user-specified suffix

[felipeek]

Any prediction of when this feature will be merged to go release?

[adamdecaf]

@felipeek It should be in Go1.11beta1 already. You can try it out!

https://groups.google.com/forum/#!topic/golang-announce/m1Szl8Mu_Fs

Also, closed issues aren't monitored. If you find a bug please open a new issue.