Skip to content

[NIO] Files.walkFileTree doesn't visit files #1519

Description

@tomwhite

There are a couple of problems running this code for deleting a directory recursively: https://github.com/HadoopGenomics/Hadoop-BAM/blob/master/src/main/java/org/seqdoop/hadoop_bam/util/NIOFileUtil.java#L63-L74

  1. The code for visitFile is never called.
  2. Files.delete(dir) fails with "Can't perform I/O on pseudo-directories (trailing slash)"

It would be good to add tests for Files.walkFileTree (and Files.walk) to the unit tests.

Activity

  1. tomwhite commented on Feb 20, 2017

    @tomwhite
    Author

    #1630 addresses some of this, but there's still a remaining issue.

    The following prints the files in a directory:

    Path p = CloudStorageFileSystem.forBucket("bucket").getPath("dir/");
    Files.list(p).forEach(System.out::println);

    However, this doesn't work:

    Path p = Paths.get(URI.create("gs://bucket/dir/"))
    Files.list(p).forEach(System.out::println);

    The problem is to do with the underlying path not having a the leading slash removed in the second case. This seems to fix the problem:

    --- a/google-cloud-contrib/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/CloudStorageFileSystemProvider.java
    +++ b/google-cloud-contrib/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/CloudStorageFileSystemProvider.java
    @@ -608,7 +608,7 @@ public final class CloudStorageFileSystemProvider extends FileSystemProvider {
         final CloudStoragePath cloudPath = CloudStorageUtil.checkPath(dir);
         checkNotNull(filter);
         initStorage();
    -    String prefix = cloudPath.toString();
    +    String prefix = cloudPath.toRealPath().toString();
         final Iterator<Blob> blobIterator = storage.list(cloudPath.bucket(),
             Storage.BlobListOption.prefix(prefix), Storage.BlobListOption.currentDirectory(),
             Storage.BlobListOption.fields()).iterateAll();
    
  2. jean-philippe-martin commented on Feb 22, 2017

    @jean-philippe-martin

    Thank you @tomwhite for the clear bug report and fix! I put together PR #1644 with the fix.

  3. jean-philippe-martin commented on Apr 17, 2017

    @jean-philippe-martin

    #1644 was merged in so this bug should be fixed now.

  4. jean-philippe-martin commented on May 15, 2017

    @jean-philippe-martin

    Whoever has the permissions to do it, please close the bug. @tomwhite please reopen if the fix doesn't work for you.

  5. jean-philippe-martin commented on Jun 12, 2017

    @jean-philippe-martin

    Thank you!

  6. added a commit that references this issue on Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions