Optimize ReadDirPlus for large directory listings

[wadhwanisimran442-cloud]

Please ensure your PR title follows the format:

type(scope): subject

Example:
feat(api): add user login endpoint

Available types:

• feat: A new feature
• fix: A bug fix
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• refactor: A code change that neither fixes a bug nor adds a feature
• perf: A code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
• ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit

Description

Link to the issue in case of a bug fix.

Testing details

1. Manual - NA
2. Unit tests - NA
3. Integration tests - NA

Any backward incompatible change? If so, please explain.

[github-actions]

Hey there and thank you for opening this pull request! 👋🏼

We require pull request titles to follow the Conventional Commits specification and it looks like your proposed title needs to be adjusted.

Details:

No release type found in pull request title "Optimize ReadDirPlus for large directory listings". Add a prefix to indicate what kind of release this pull request corresponds to. For reference, see https://www.conventionalcommits.org/

Available types:
 - feat: A new feature
 - fix: A bug fix
 - docs: Documentation only changes
 - style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
 - refactor: A code change that neither fixes a bug nor adds a feature
 - perf: A code change that improves performance
 - test: Adding missing tests or correcting existing tests
 - build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
 - ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
 - chore: Other changes that don't modify src or test files
 - revert: Reverts a previous commit

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces performance optimizations for directory listings in large buckets by implementing a streaming mechanism for ReadDirPlus. By moving away from loading entire directory contents into memory, the file system can now handle significantly larger directories without hitting memory constraints. The changes include refactoring the directory handle to support pagination, decoupling it from the core file system logic, and ensuring robust handling of inode lookups during the streaming process.

Highlights

• Streaming ReadDirPlus: Implemented a streaming approach for ReadDirPlus to handle large directory listings efficiently by fetching entries in pages from GCS instead of loading all entries into memory.
• Refactored DirHandle: Updated DirHandle to manage streaming state, including GCS markers and buffered entries, and decoupled it from the main file system implementation using a new FileSystem interface.
• Inode Lookup Optimization: Improved inode handling during directory listing by introducing a more robust lookup mechanism that handles potential object deletions between listing and attribute fetching.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request introduces a streaming implementation for ReadDirPlus to improve performance and memory efficiency when listing large directories. It also includes refactoring to break direct dependencies between the directory handle and the file system, along with minor bug fixes in temporary file handling and configuration mapping. However, the new ReadDirPlus implementation currently ignores local unsynced entries, which is a regression from the previous buffered logic. Additionally, there is redundant type assertion logic in the root inode initialization and a change in error handling strategy during directory listing that could lead to silent failures.

[gemini-code-assist]

The new streaming implementation of ReadDirPlus currently ignores localEntries (files created locally but not yet synced to GCS). This is a regression in functionality compared to the previous buffered implementation, which merged local entries into the listing. This means that newly created or modified files that haven't been flushed will be missing from ReadDirPlus results, leading to inconsistencies in directory listings (e.g., ls might show a file that ls -l misses).

[gemini-code-assist]

The type assertion for root as inode.BucketOwnedDirInode is performed twice consecutively. The outer if check at line 299 makes the inner check at line 301 redundant. Additionally, the else block at line 307 is unreachable because if the outer check passes, the inner one will also pass. The logic can be simplified by combining these checks.

	if boDi, ok := root.(inode.BucketOwnedDirInode); ok {
		if boDi.Bucket().BucketType().Hierarchical {
			fs.folderInodes[root.Name()] = root
		} else {
			fs.implicitDirInodes[root.Name()] = root
		}
	} else {
		fs.implicitDirInodes[root.Name()] = root
	}

[gemini-code-assist]

In the new ReadDirPlus implementation, errors encountered during coreToDirentPlus are logged as warnings and the entry is skipped. The previous implementation in fs.go would return the error immediately, failing the ReadDirPlus operation. Swallowing these errors might lead to incomplete directory listings without the caller knowing that an error occurred. It is generally better to propagate the error to the VFS layer unless skipping the entry is the intended behavior for specific, recoverable errors.