os: hard to use DirFS for cross-platform root filesystem

[josharian]

I wrote a little package called cdup that walks a directory and each of its ancestors looking for a file/dir. This is useful e.g. to find the root of a git dir (look for .git) or a Go module (look for go.mod).

I aimed to write it using fs.FS and use os.DirFS as a bridge to the OS filesystem as needed, perhaps inside a wrapper function.

I may just be holding it wrong, but I encountered a few surprising bumps.

A typical invocation is cdup.Find(pwd, ".git"). The obvious but wrong translation of this is cdup.FindIn(os.DirFS("/"), pwd, ".git").

What is the correct root path to pass to os.DirFS on Windows? filepath.VolumeName seems like the right API to use. (I think?)

Then we must convert pwd to a relative path using filepath.Rel. On Windows, we also need to call filepath.ToSlash.

So I believe that the correct code is something like:

root := "/"
if vol := filepath.VolumeName(dir); vol != "" {
  root = vol
}
rel := filepath.Rel(root, dir)
rel = filepath.ToSlash(rel)
cdup.FindIn(os.DirFS(root), rel, ".git")

This is a surprising amount of non-obvious work for what seems like it should be a trivial bridge between fs.FS and the OS filesystem.

I'm not sure exactly what the fix should be here. Docs/examples? New package filepath API to get the OS fs root for a path? New package os API to construct a root filesystem (and expose the root for use with filepath.Rel)?

[dmitshur]

CC @rsc.

[bcmills]

What is the correct root path to pass to os.DirFS on Windows? filepath.VolumeName seems like the right API to use. (I think?)

IIUC there is no single root path on Windows: each drive letter is its own root.

Perhaps you could implement an fs.FS that identifies all mapped (or all local) drive letters and lists those as virtual subdirectories of the root?

[josharian]

At least for my purposes, I don't need a single root. I only need the root corresponding to a particular path, since I'm only walking up the tree. The broader question is interesting too, though.

(Ideally, for my purposes, I'd like to stop at filesystem boundaries, instead of traversing through mount points. But that's out of scope—the point of this issue is that it's hard to do something that feels like it should be easy. It is reasonable for detecting and stopping at mount points to be hard. 😁)

[jwalton]

I think the real problem here is, if you want to write an application that can open any file on the filesystem, then on Linux/Mac you can do:

files := os.DirFS("/")

but on Windows, this will make your application unusable:

files := os.DirFS("/")
dir, err := fs.ReadDir(files, "c:/Windows/win.ini") // Error open c:/Windows/win.ini: invalid argument
dir, err := fs.ReadDir(files, "Windows/win.ini") // Error open //Windows/win.ini: The network path was not found

"The network path was not found" is especially weird, as it suggests that fs.ReadDir(files, "sambahost/foo/bar") will work, which is probably not what anyone expects.

On Windows, you need to work out which volume the file is on and modify the path before you can even call DirFS. If I have a function doTheThing(files fs.FS, path string), and I want to call it correctly in a cross-platform way for an arbitrary file, I need all of that boilerplate from @josharian's example every time I call this function.

I suspect what this means is that many Go app authors will forget about the Windows case and do files := os.DirFS("/"), and the app will just fail to run correctly on Windows machines. It seems like os.DirFS("") on Windows is roughly equivalent to os.DirFS(filepath.VolumeName(cwd)), so if you do os.DirFS("") it'll have a chance of working on Windows machines where CWD and the file you want to open are on the same volume, although you'ld still have to strip the volume from the absolute filename.

As a library author, the obivous solution here is to write your function as DoTheThing(path string), and do these gymnastics inside your library... but at that point there seems little benefit to using fs.FS in the first place. There's no way to pass a different fs.FS implementation to DoTheThing(), so you can't pass in a fake testfs instance anyways. At that point you may as well just use os.Open().

It would be really nice if this:

files := os.DirFS("")
file, err := files.Open('d:/foo/bar.txt')

did what most people probably expect it to do. Then you'd still need to call filepath.ToSlash(), but at least you could open an arbitrary file without writing a short novel every time, but we probably don't want to change the existing behavior. Some app somewhere probably relies on that...

[jwalton]

Maybe we could add a os.SystemFS() that behaves like this:

files := os.SystemFS()

// On Windows
file, err := files.Open('d:/foo/bar.txt')
file, err := files.Open('//networkShare/foo/bar')

On Linux
file, err := files.Open('/foo/bar')

[jwalton]

Actually - one more possible solution. On Windows, we could make it so os.DirFS("/") returns a FS instance that accepts "d:/foo/bar.txt" or "//networkShare/foo/bar":

files := os.DirFS("/")

// On Windows
file, err := files.Open('d:/foo/bar.txt')
file, err := files.Open('//networkShare/foo/bar')

On Linux
file, err := files.Open('/foo/bar')

This changes the behavior of os.DirFS(), but since os.DirFS("/") currently returns a more-or-less unusable FS instance on Windows it's unlikely anyone relies on it right now. Also people seem to think this should work anyways, so this fixes apps that are likely already broken.

[qmuntal]

I've also been hit by this issue in production code, and after several attempts to fix it I just revert the code to use good old filepath+os.

I would be happy with a safe cross-platform example, but a new os API will be also welcomed.

[bcmills]

file, err := files.Open('/foo/bar')

Note that io/fs.FS.Open supports only unrooted paths (see https://pkg.go.dev/io/fs#ValidPath), so /foo/bar is not valid on Linux today.

[jwalton]

Ah, my bad:

files := os.DirFS("/")

// On Windows
file, err := files.Open('d:/foo/bar.txt')
file, err := files.Open('networkShare/foo/bar')

On Linux
file, err := files.Open('foo/bar')

This should be a reatively easy and safe change to make, since two out of three of these examples already work, and the third today returns an error, and it also seems pretty clear that this was probably what the caller intended.

[bcmills]

I don't think os.DirFS("/") itself should support UNC network shares — the equivalent behavior on Unix filesystems does not enumerate network shares unless the root filesystem itself includes mounts of all of those shares (which seems unlikely).

I think the less surprising behavior would be a filesystem that resembles the WSL2 /mnt directory, with a relative directory (without a colon) for each mapped drive letter:

files := os.DirFS("/")
file, err := files.Open("d/foo/bar.txt")