feat(services/gdrive): List shows modified timestamp gdrive #5226

erickguan · 2024-10-22T11:36:12Z

Which issue does this PR close?

Part of #4746.

Are there any user-facing changes?

None.

erickguan · 2024-10-22T19:14:56Z

I did some experimentation (patch) with async support for listing operations, which helps reduce runtime as expected. However, extending OpList introduces a "breaking" change. Since OpenDAL has a compatibility package, I’m happy to coordinate with you when you're preparing for a breaking release, to minimize the amount of work on both sides. There's no urgency regarding async support, though.

A few other observations from running the behavior tests (not related to the PR):

The behavior test fails on my local machine but passes in CI. I’m investigating the cause casually.
The UUID random generation in behavior tests creates multiple levels of UUIDs. While this results in an extremely low chance of collision in the CI for OpenDAL's test accounts, I’d like to reduce the number of levels when running against my environment too.

Xuanwo · 2024-10-23T02:50:23Z

core/src/services/gdrive/lister.rs

        // Return self at the first page.
        if ctx.token.is_empty() && !ctx.done {
            let path = build_rel_path(&self.core.root, &self.path);
-            let e = oio::Entry::new(&path, Metadata::new(EntryMode::DIR));
+            let mut metadata = Metadata::new(EntryMode::DIR);
+            if stat_file_metadata {


Hi, sorry for not making the Metakey behavior clearer. (I'm working on this.)

Metakey represents a best effort hint and is not processed server-side. The service merely needs to supply the most comprehensive metadata available during listing.

The Operator will call stat as required, for example:

opendal/core/src/types/list.rs

Lines 150 to 211 in 5e074cc

fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {

// Returns `None` if we have errored.

if self.errored {

return Poll::Ready(None);

}

// Trying to pull more tasks if there are more space.

if self.tasks.has_remaining() {

// Building future if we have a lister available.

if let Some(mut lister) = self.lister.take() {

let fut = async move {

let res = lister.next_dyn().await;

(lister, res)

};

self.fut = Some(Box::pin(fut));

}

if let Some(fut) = self.fut.as_mut() {

if let Poll::Ready((lister, entry)) = fut.as_mut().poll(cx) {

self.lister = Some(lister);

self.fut = None;

match entry {

Ok(Some(oe)) => {

let (path, metadata) = oe.into_entry().into_parts();

if metadata.contains_metakey(self.required_metakey) {

self.tasks

.push_back(StatTask::Known(Some((path, metadata))));

} else {

let acc = self.acc.clone();

let fut = async move {

let res = acc.stat(&path, OpStat::default()).await;

(path, res.map(|rp| rp.into_metadata()))

};

self.tasks.push_back(StatTask::Stating(Box::pin(fut)));

}

}

Ok(None) => {

self.lister = None;

}

Err(err) => {

self.errored = true;

return Poll::Ready(Some(Err(err)));

}

}

}

}

}

// Try to poll tasks

if let Some((path, rp)) = ready!(self.tasks.poll_next_unpin(cx)) {

let metadata = rp?;

return Poll::Ready(Some(Ok(Entry::new(path, metadata))));

}

if self.lister.is_some() || self.fut.is_some() {

Poll::Pending

} else {

Poll::Ready(None)

}

}

}

Thanks for the review. I get a weird Metadata instance problem, that Gdrivebackend::stat returns metadata with a timestamp but Lister::poll_next gets a result of Metadata without the timestamp.

I will debug it a bit.

erickguan · 2024-10-27T19:13:22Z

@Xuanwo Thanks for the pointer. I've figured out how the PageList and Google Drive backend work together, but I’m currently stuck on a failing test with MinIO in the complete layer. Am I on the right track in continuing this PR?

From what I understand, the `CompleteLayer`` depends on the behavior of different services. Since you mentioned that work on metakey is underway, I’m trying not to alter OpenDAL’s behavior. Unfortunately, adding the new stat behavior in this layer breaks the MinIO service.

Xuanwo · 2024-11-05T02:14:40Z

core/src/layers/complete.rs

@@ -174,7 +174,12 @@ impl<A: Access> CompleteAccessor<A> {
        }

        if path == "/" {
-            return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
+            let meta = if capability.stat {
+                self.inner.stat(path, args).await?.into_metadata()


Hi, it's intended that we don't send stat for /. The service should always make sure / exists.

Xuanwo · 2024-11-05T02:19:00Z

core/src/services/gdrive/lister.rs

+            let normalized_path = build_rel_path(root, &path);
+
+            let mut meta = Metadata::new(file_type);
+            // Resetting the metakey is necessary when mode is `EntryMode::DIR`,


The underlying cause might be here:

opendal/core/src/types/metadata.rs

Lines 58 to 61 in 259a19e

// If mode is dir, we should always mark it as complete.

if mode.is_dir() {

metakey |= Metakey::Complete

}

It seems reasonable to remove it.

Xuanwo · 2024-11-06T03:22:40Z

Also related to #5287

erickguan · 2024-11-06T11:02:50Z

Thanks for tentative checking in with metakey!

erickguan · 2024-11-06T18:17:24Z

After a closer look, can I pause this PR until your metakey work?

Xuanwo · 2024-11-07T03:54:07Z

After a closer look, can I pause this PR until your metakey work?

Yes, after the metakey is refactored, our work here will be much cleaner.

Xuanwo · 2024-11-14T16:13:46Z

Yes, after the metakey is refactored, our work here will be much cleaner.

Hi, you can continue your work now.

erickguan · 2024-11-14T16:18:52Z

Thanks for the ping! I'll pick it up and ping you when it's ready.

Xuanwo · 2024-11-14T16:23:05Z

Thanks for the ping! I'll pick it up and ping you when it's ready.

Thank you for your patience. Looking forward to your PR.

Xuanwo

Thank you, @erickguan, this PR looks mostly good to me now!

Xuanwo · 2024-11-19T16:18:21Z

core/src/layers/complete.rs

@@ -177,8 +177,19 @@ impl<A: Access> CompleteAccessor<A> {
            return Ok(RpStat::new(Metadata::new(EntryMode::DIR)));
        }

-        // Forward to inner if create_dir is supported.
-        if path.ends_with('/') && capability.create_dir {
+        if path.ends_with('/')


Hi, let's keep this file unchanged. The create_dir capability check is related to special handling for object storage. I will separate this part to make it clearer.

Here is the problem. The Google Drive service supports create_dir, so OpenDAL will remove the metadata from the Google Drive service without this change.

// Forward to inner if create_dir is supported. if path.ends_with('/') && capability.create_dir { let meta = self.inner.stat(path, args).await?.into_metadata(); if meta.is_file() { return Err(Error::new( ErrorKind::NotFound, "stat expected a directory, but found a file", )); } return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); }

I read the RFC 3243, and I am skeptical of the change in this PR. I am not familiar with the context so I have a few ideas in mind:

Add a comment explaining the hack to clean up later.

Maybe extend the complete layer to respond meta without creating a new one.

Oh, I see. Let's return the metadata we fetched from storage directly.

Ok. Happy to defer this problem to you!

core/src/layers/complete.rs

Xuanwo

Thank you @erickguan for this PR and your patience!

github-actions bot added the releases-note/feat label Oct 22, 2024

erickguan force-pushed the modified-timestamp-gdrive branch 3 times, most recently from 77cf29b to 0fde283 Compare October 22, 2024 18:48

erickguan marked this pull request as ready for review October 22, 2024 19:14

erickguan requested a review from Xuanwo as a code owner October 22, 2024 19:14

Xuanwo reviewed Oct 23, 2024

View reviewed changes

erickguan requested a review from Xuanwo October 27, 2024 19:14

Xuanwo reviewed Nov 5, 2024

View reviewed changes

erickguan force-pushed the modified-timestamp-gdrive branch 3 times, most recently from f22ee2b to 604916d Compare November 19, 2024 16:20

Xuanwo reviewed Nov 20, 2024

View reviewed changes

core/src/layers/complete.rs Show resolved Hide resolved

erickguan added 2 commits November 20, 2024 19:50

feat(core/services-gdrive): return stat metadata result

6c0d2a6

fixup! feat(core/services-gdrive): return stat metadata result

44823a2

erickguan force-pushed the modified-timestamp-gdrive branch from 222ddd5 to 44823a2 Compare November 20, 2024 18:50

erickguan requested a review from Xuanwo November 20, 2024 19:07

Xuanwo approved these changes Nov 21, 2024

View reviewed changes

Xuanwo merged commit e5a4104 into apache:main Nov 21, 2024
239 checks passed

erickguan deleted the modified-timestamp-gdrive branch November 21, 2024 13:34

erickguan mentioned this pull request Dec 1, 2024

bug: multiple file with same name in google drive crash #5099

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(services/gdrive): List shows modified timestamp gdrive #5226

feat(services/gdrive): List shows modified timestamp gdrive #5226

erickguan commented Oct 22, 2024 •

edited

Loading

erickguan commented Oct 22, 2024 •

edited

Loading

Xuanwo Oct 23, 2024

erickguan Oct 23, 2024

erickguan commented Oct 27, 2024 •

edited

Loading

Xuanwo Nov 5, 2024

Xuanwo Nov 5, 2024

Xuanwo commented Nov 6, 2024

erickguan commented Nov 6, 2024

erickguan commented Nov 6, 2024

Xuanwo commented Nov 7, 2024

Xuanwo commented Nov 14, 2024

erickguan commented Nov 14, 2024

Xuanwo commented Nov 14, 2024

Xuanwo left a comment

Xuanwo Nov 19, 2024

erickguan Nov 20, 2024 •

edited

Loading

Xuanwo Nov 20, 2024

erickguan Nov 20, 2024

Xuanwo left a comment

	fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
	// Returns `None` if we have errored.
	if self.errored {
	return Poll::Ready(None);
	}

	// Trying to pull more tasks if there are more space.
	if self.tasks.has_remaining() {
	// Building future if we have a lister available.
	if let Some(mut lister) = self.lister.take() {
	let fut = async move {
	let res = lister.next_dyn().await;
	(lister, res)
	};
	self.fut = Some(Box::pin(fut));
	}

	if let Some(fut) = self.fut.as_mut() {
	if let Poll::Ready((lister, entry)) = fut.as_mut().poll(cx) {
	self.lister = Some(lister);
	self.fut = None;

	match entry {
	Ok(Some(oe)) => {
	let (path, metadata) = oe.into_entry().into_parts();
	if metadata.contains_metakey(self.required_metakey) {
	self.tasks
	.push_back(StatTask::Known(Some((path, metadata))));
	} else {
	let acc = self.acc.clone();
	let fut = async move {
	let res = acc.stat(&path, OpStat::default()).await;
	(path, res.map(\|rp\| rp.into_metadata()))
	};
	self.tasks.push_back(StatTask::Stating(Box::pin(fut)));
	}
	}
	Ok(None) => {
	self.lister = None;
	}
	Err(err) => {
	self.errored = true;
	return Poll::Ready(Some(Err(err)));
	}
	}
	}
	}
	}

	// Try to poll tasks
	if let Some((path, rp)) = ready!(self.tasks.poll_next_unpin(cx)) {
	let metadata = rp?;
	return Poll::Ready(Some(Ok(Entry::new(path, metadata))));
	}

	if self.lister.is_some() \|\| self.fut.is_some() {
	Poll::Pending
	} else {
	Poll::Ready(None)
	}
	}
	}

	// If mode is dir, we should always mark it as complete.
	if mode.is_dir() {
	metakey \|= Metakey::Complete
	}

feat(services/gdrive): List shows modified timestamp gdrive #5226

feat(services/gdrive): List shows modified timestamp gdrive #5226

Conversation

erickguan commented Oct 22, 2024 • edited Loading

Which issue does this PR close?

Are there any user-facing changes?

erickguan commented Oct 22, 2024 • edited Loading

Xuanwo Oct 23, 2024

Choose a reason for hiding this comment

erickguan Oct 23, 2024

Choose a reason for hiding this comment

erickguan commented Oct 27, 2024 • edited Loading

Xuanwo Nov 5, 2024

Choose a reason for hiding this comment

Xuanwo Nov 5, 2024

Choose a reason for hiding this comment

Xuanwo commented Nov 6, 2024

erickguan commented Nov 6, 2024

erickguan commented Nov 6, 2024

Xuanwo commented Nov 7, 2024

Xuanwo commented Nov 14, 2024

erickguan commented Nov 14, 2024

Xuanwo commented Nov 14, 2024

Xuanwo left a comment

Choose a reason for hiding this comment

Xuanwo Nov 19, 2024

Choose a reason for hiding this comment

erickguan Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Xuanwo Nov 20, 2024

Choose a reason for hiding this comment

erickguan Nov 20, 2024

Choose a reason for hiding this comment

Xuanwo left a comment

Choose a reason for hiding this comment

erickguan commented Oct 22, 2024 •

edited

Loading

erickguan commented Oct 22, 2024 •

edited

Loading

erickguan commented Oct 27, 2024 •

edited

Loading

erickguan Nov 20, 2024 •

edited

Loading