Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new: Added jina clip text embedding #408

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

hh-space-invader
Copy link
Contributor

@hh-space-invader hh-space-invader commented Nov 19, 2024

Adding jinaai/jina-clip-v1
They provided two examples, the first one works and the second one complains about missing jinaai/jina-clip-v1/sentence_xlnet_config.json. The output of the first one seems to have small numbers, like they are normallized but its not mentioned so not sure tbh.

Update:
The text model needs pooling and normalizing
The image model needs the image to be square

All Submissions:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  • Does your submission pass the existing tests?
  • Have you added tests for your feature?
  • Have you installed pre-commit with pip3 install pre-commit and set up hooks with pre-commit install?

New models submission:

  • Have you added an explanation of why it's important to include this model?
  • Have you added tests for the new model? Were canonical values for tests computed via the original model?
  • Have you added the code snippet for how canonical values were computed?
  • Have you successfully ran tests with your changes locally?

@hh-space-invader hh-space-invader changed the title WIP: Added jina clip text embedding new: Added jina clip text embedding Nov 21, 2024
Comment on lines +127 to +139
def resize2square(
image: Image.Image,
size: int,
fill_color: Optional[Union[str, int, tuple[int, ...]]] = None,
resample: Union[Image.Resampling, int] = Image.Resampling.BICUBIC,
) -> Image.Image:
resized_image = resize(image=image, size=size, resample=resample)

new_image = Image.new(mode="RGB", size=(size, size), color=fill_color)
left = (size - resized_image.size[0]) // 2
top = (size - resized_image.size[1]) // 2
new_image.paste(resized_image, (left, top))
return new_image
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have resize, let's not introduce new functions, we can add an additional parameter like preserve_aspect_ratio or something like this to resize.
If it's false then we should just resize image to the required size (preserving aspect ratio is useful when later we have crop)

Comment on lines +227 to +233
if config.get("do_normalize", False) or ("mean" in config and "std" in config):
transforms.append(
Normalize(
mean=config.get("image_mean", config.get("mean")),
std=config.get("image_std", config.get("std")),
)
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if config.get("do_normalize", False) or ("mean" in config and "std" in config):
transforms.append(
Normalize(
mean=config.get("image_mean", config.get("mean")),
std=config.get("image_std", config.get("std")),
)
)
if config.get("do_normalize", False):
transforms.append(Normalize(mean=config["image_mean"], std=config["image_std"]))
elif "mean" in config and "std" in config:
transforms.append(Normalize(mean=config["mean"], std=config["std"]))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants