Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pimo tutorials/04 advanced iv #2352

Merged

Conversation

jpcbertoldo
Copy link
Contributor

📝 Description

add notebook 701e_aupimo_advanced_iv

teaches how to load/save from the standard format implemented in https://github.com/jpcbertoldo/aupimo (official aupimo repo)

shows how to perform statistical comparisons between two models ("how significant is the difference between these two models?" or "is model A consistently better than B? (or just by chance?)"

will be opened after #2348 is merged

✨ Changes

Select what type of change your PR is:

  • 🐞 Bug fix (non-breaking change which fixes an issue)
  • 🔨 Refactor (non-breaking change which refactors the code base)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

  • 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
  • 📚 I have made the necessary updates to the documentation (if applicable).
  • 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@jpcbertoldo jpcbertoldo changed the title [WIP] Pimo tutorials/04 advanced iv Pimo tutorials/04 advanced iv Oct 8, 2024
Copy link
Contributor

@samet-akcay samet-akcay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good, thanks

…parisons

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
@jpcbertoldo jpcbertoldo marked this pull request as ready for review October 15, 2024 15:00
@samet-akcay samet-akcay enabled auto-merge (squash) October 15, 2024 15:29
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
auto-merge was automatically disabled October 16, 2024 12:16

Head branch was pushed to by a user without write access

@jpcbertoldo
Copy link
Contributor Author

@samet-akcay can you trigger the CI here please?

@jpcbertoldo
Copy link
Contributor Author

The coverage failure is from the missing gpu issue again

    @staticmethod
    def _choose_gpu_accelerator_backend() -> str:
        if MPSAccelerator.is_available():
            return "mps"
        if CUDAAccelerator.is_available():
            return "cuda"
>       raise MisconfigurationException("No supported gpu backend found!")
E       lightning.fabric.utilities.exceptions.MisconfigurationException: No supported gpu backend found!

@jpcbertoldo
Copy link
Contributor Author

The coverage failure is from the missing gpu issue again

    @staticmethod
    def _choose_gpu_accelerator_backend() -> str:
        if MPSAccelerator.is_available():
            return "mps"
        if CUDAAccelerator.is_available():
            return "cuda"
>       raise MisconfigurationException("No supported gpu backend found!")
E       lightning.fabric.utilities.exceptions.MisconfigurationException: No supported gpu backend found!

Hey i think this is mergeable but the auto-merge is blocked because of this coverage

@samet-akcay
Copy link
Contributor

argh, CI machine is to be restarted :/

Copy link

codecov bot commented Oct 17, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.76%. Comparing base (0823ab8) to head (d4975b7).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2352      +/-   ##
==========================================
+ Coverage   80.75%   80.76%   +0.01%     
==========================================
  Files         255      255              
  Lines       11496    11497       +1     
==========================================
+ Hits         9284     9286       +2     
+ Misses       2212     2211       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@samet-akcay samet-akcay merged commit 1465b05 into openvinotoolkit:main Oct 17, 2024
7 checks passed
cjy513203427 pushed a commit to cjy513203427/anomalib_forked that referenced this pull request Oct 22, 2024
* add notebook 701e_aupimo_advanced_iv on load/save and statistical comparisons

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* make `AUPIMOResult.num_thresholds` optional

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add aupimo notebook advanced iv (load/save and statistical tests)

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* simplify cite us and mention intal

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* fix readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>
samet-akcay added a commit that referenced this pull request Nov 15, 2024
* Update timm requirement from <=1.0.7,>=1.0.7 to >=1.0.7,<=1.0.9 (#2274)

* Update timm requirement from <=1.0.7,>=1.0.7 to >=1.0.7,<=1.0.9

Updates the requirements on [timm](https://github.com/huggingface/pytorch-image-models) to permit the latest version.
- [Release notes](https://github.com/huggingface/pytorch-image-models/releases)
- [Commits](huggingface/pytorch-image-models@v1.0.7...v1.0.9)

---
updated-dependencies:
- dependency-name: timm
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update pyproject.toml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* 🐞Update `setuptools` requirement for PEP 660 support (#2320)

Update setup tools

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Fix transforms for draem, dsr and rkde (#2324)

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Add check before loading metrics data from checkpoint (#2323)

Add check before loading from checkpoint

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Add PIMO (#2329)

* PIMO (#1726)

* update

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* test binclf curves numpy and numba and fixes

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct som docstrings

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* torch interface and tests

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* torch interface and tests

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* constants regrouped in dataclass as class vars

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* result class was unneccesary for per_image_binclf_curve

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* factorize function _get_threshs_minmax_linspace

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* small docs fixes

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add pimo numpy version and test

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* move validation

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add `shared_fpr_metric` option

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add pimo torch functional version and test

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add torchmetrics interface and test

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* renames and put things in init

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* validate inputs in result objects

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* result objects to from dict and tests

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add save and load methods to result objects and test

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor validations and minor changes

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* test result objects' properties

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* minor refactors

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add missing docstrings

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* minore vocabulary fix for consistency

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add per image scores statistics and test it

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor constants notation

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add stats tests and test it

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* change the meaning of AUPIMO.num_thresh

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* interface to format pairwise test results

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* improve doc

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add optional `paths` to result objects and some minor fixes and refactors

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove frozen from dataclasses and some done todos

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* review headers

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* doc modifs

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor `score_less_than_thresh` in `_binclf_one_curve_python`

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct license comments

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* fix doc

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* numba as extra requirement

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor copyrights from jpcbertoldo

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove from __future__ import annotations

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor validations names

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* dedupe file path validation

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* fix tests

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Add todo

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor enums

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* only logger.warning

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor test imports

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor docs

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* refactor some docs

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct pre commit errors

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove author tag

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add thrid party program

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Update src/anomalib/metrics/per_image/pimo.py

* move HAS_NUMBA

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* remove PIMOSharedFPRMetric

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* make torchmetrics compute avg by dft

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* pre-commit hooks corrections

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct numpy.trapezoid

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* 🗑️ Remove numba (#2313)

* remove numba

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* fix pre-commit checks

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* add third-party-programs.txt

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* 🗑️ Remove unused methods (#2315)

* remove numba

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* fix pre-commit checks

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* remove all unused methods

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* PIMO: Port Numpy → Torch (#2316)

* remove numba

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* fix pre-commit checks

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* remove all unused methods

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* replace numpy with torch

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* 🔨Refactor methods across files (#2321)

* remove numba

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* fix pre-commit checks

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* remove all unused methods

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* replace numpy with torch

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* refactor code

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* refactor
move functional inside update
remove path from the metric

* Add changes from comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Remove model to model comparison (#2325)

* rename to pimo

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* minor refactor

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* remove model to model comparison

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* fix test

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* PR comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Minor refactor

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* PR comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Remove unused enums

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* update doc strings

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* update param names

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* add aupimo basic usage tutorial notebook (#2330)

* add aupimo basic usage tutorial notebook

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* update scipy import

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add cite us

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* minor

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* modify texts and add illustration

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* udpate working dir

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>
Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Makes batch size dynamic (#2339)

Made batch dimension of ONNX export dynamic when specifying input shape.

* Add pimo tutorial advanced i (fixed) (#2336)

* uset all padim features to make it deterministic

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add aupimo notebook advanced i

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* update readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* modify changelog

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct again

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* minor corrections

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Pimo tutorials/02 advanced ii (#2347)

* uset all padim features to make it deterministic

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add aupimo notebook advanced i

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* update readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* modify changelog

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct again

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* minor corrections

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add aupimo notebook advanced ii (pimo curve and integration bounds)

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* fix links

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* correct change log

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* Create epic.yaml

* 🔨 Update the issue templates (#2363)

* Update epic.yaml

* Update epic.yaml

* Update epic.yaml

* Update epic.yaml

* Update task.yaml

* Create user_story.yaml

* Update epic.yaml

* Pimo tutorials/03 advanced iii (#2348)

* add aupimo notebook advanced iii (aupimo score of a random model)

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add cite us

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* update notebooks readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* 🔨 Deprecate try import and replace it with Lightning's package_available (#2373)

Replace try_import with lightnings package_available function

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Refactor folder3d to avoid complex-structure (C901) issue (#2185)

* Refactored-make_folder3d_dataset-ruff-error-C901 (#1926)

Signed-off-by: sahusiddharth <siddharth.sahu@plaksha.edu.in>

* Simplify folder 3d dataset (#2184)

---------

Signed-off-by: sahusiddharth <siddharth.sahu@plaksha.edu.in>
Co-authored-by: Siddharth Sahu <112792547+sahusiddharth@users.noreply.github.com>

* 🚀 Add datumaro annotation dataloader (#2377)

* Add datumaro annotation dataloader

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Update changelog

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add examples

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Pimo tutorials/04 advanced iv  (#2352)

* add notebook 701e_aupimo_advanced_iv on load/save and statistical comparisons

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* make `AUPIMOResult.num_thresholds` optional

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* add aupimo notebook advanced iv (load/save and statistical tests)

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* simplify cite us and mention intal

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

* fix readme

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>

---------

Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* 🐞 Defer OpenVINO import to avoid unnecessary warnings (#2385)

* Fix openvino import issue

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Fix pre-commit issues

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

---------

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* 🚀 Add VLM based Anomaly Model (#2344)

* [Draft] Llm on (#2165)

* Add TaskType Explanation

Signed-off-by: Bepitic <bepitic@gmail.com>

* Add llm model

Signed-off-by: Bepitic <bepitic@gmail.com>

* add ollama

Signed-off-by: Bepitic <bepitic@gmail.com>

* better description for descr in title

Signed-off-by: Bepitic <bepitic@gmail.com>

* add text of llm into imageResult visualization

* add text of llm into imageResult visualization

Signed-off-by: Bepitic <bepitic@gmail.com>

* latest changes

Signed-off-by: Bepitic <bepitic@gmail.com>

* add wip llava/llava_next

Signed-off-by: Bepitic <bepitic@gmail.com>

* add init

Signed-off-by: Bepitic <bepitic@gmail.com>

* add text of llm into imageResult visualization

Signed-off-by: Bepitic <bepitic@gmail.com>

* latest changes

Signed-off-by: Bepitic <bepitic@gmail.com>

* upd Lint

Signed-off-by: Bepitic <bepitic@gmail.com>

* fix visualization with description

Signed-off-by: Bepitic <bepitic@gmail.com>

* show the images every batch

Signed-off-by: Bepitic <bepitic@gmail.com>

* fix docstring and error management

Signed-off-by: Bepitic <bepitic@gmail.com>

* Add compatibility for TaskType.EXPLANATION.

Signed-off-by: Bepitic <bepitic@gmail.com>

* Remove, show in the engine-Visualization.

* fix visualization and llm openai multishot.

* fix Circular import problem

* Add HugginFace To LLavaNext

Signed-off-by: Bepitic <bepitic@gmail.com>

---------

Signed-off-by: Bepitic <bepitic@gmail.com>

* 🔨 Scaffold for refactor (#2340)

* initial scafold

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Apply PR comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* rename dir

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add ChatGPT (#2341)

* initial scafold

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Apply PR comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* rename dir

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* delete llm_ollama

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add ChatGPT

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add ChatGPT

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Remove LLM model

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add Huggingface (#2343)

* initial scafold

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Apply PR comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* rename dir

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* delete llm_ollama

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add ChatGPT

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add ChatGPT

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Remove LLM model

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add transformers

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Remove llava

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* 🔨 Minor Refactor (#2345)

Refactor

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* undo changes

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* undo changes

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* undo changes to image.py

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Add explanation visualizer (#2351)

* Add explanation visualizer

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* bug-fix

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* 🔨 Allow setting API keys from env (#2353)

Allow setting API keys from env

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* 🧪 Add tests (#2355)

* Add tests

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* remove explanation task type

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* minor fixes

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Update changelog

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Fix tests

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Address PR comments

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* update name

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

* Update src/anomalib/models/image/vlm_ad/lightning_model.py

Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* update name

Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>

---------

Signed-off-by: Bepitic <bepitic@gmail.com>
Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>
Co-authored-by: Paco <bepitic@gmail.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Add ensembling methods for tiling to Anomalib (#1226)

* Fixed broken links in readme

* Fixed inference command in readme

* Add tiling for ensemble

* Add tests for tiling for ensemble

* Moved ensemble tiler to separate file

* Modify padim config for ensemble

* Add tiling to dataset

* Revert changes to train

* Add tiling to collate fn

* Fix tiling in collate

* Change val. function to protected

* Add tile number logic

* Move collate fn to separate file

* Update tests for tiler

* Add training loop for ensemble

* Add model input size setup

* Move ens config to separate file

* Revert mvtec modifications

* Remove unused imports in mvtec

* Add batch adjustment to untiling

* Add predict step to ensemble

* Add comment and docstring to tile joining function

* Move tile joining to separate function

* Add joining for all tiled data

* Add joining for all box data

* Refactor pred. joining as modular class

* Fix box joining

* Add label and score joining

* Add ensemble visualization

* Add end of predict hook

* Add metric computation

* Fix metric thresholds

* Add removal of individual visualization

* Add demo1 notebook

* Add docstrings and cleanup

* Add memory benchmark

* Add modular class for storing predictions

* Add metric to separate class

* Refactor to support prediction data class

* Rename predictions class

* Add filesystem predictions class

* Add resized predictions class

* Fix joiner for classification task

* Add page peak to memory benchmark

* Add global stats calculation

* Add docstrings to stats calculation

* Refactor joiner for pipeline

* Refactor stats into pipeline

* Refactor metrics as pipeline block

* Refactor visualization as pipeline block

* Refactor postprocessing into a pipeline

* Add normalization and thresholding on joined predictions

* Refactor tiler to accept config file

* Add smoothing of tile joins.

* Refactor ensemble datamodule preparation

* Remove unused changes in dataloader

* Fix metric configuration

* Fix box coordinates in joining

* Add ensemble callbacks preparation function

* Fix box prediction bug in postprocess

* Add ensemble params to config

* Refactor postprocessing.

* Refactor post-processing

* Refactor predictions

* Code cleanup

* Optimize prediction storage

* Make join smoothing configurable

* Cleanup before PR

* Fix stats pipeline

* Fix logging strings

* Fix memory benchmark

* Fix tiler issues

* Fix import issues

* Fix naming in metrics and visualization

* Fix cyclic import

* Make logging lazy

* Refactor tiler tests

* Added collate tiling tests

* Added ensemble helper functions tests

* Refactor for dummy ensemble config

* Refactor for dummy base config

* Add tests for prediction storage

* Add tests for prediction joiner

* Add tests for visualization

* Fix small issues in tests

* Add metrics test

* Add post-processing tests

* Fix tiler to work with different instance

* Move seed setting inside train loop

* Fix pipeline stats bug

* Rename ensemble config fixture

* Add pipeline tests

* Fix config in pipeline tests

* Add training script test

* Fix types and docstrings

* Move and rename to tiled_ensemble

* Fix bug in label joining.

* Remove memory benchmark

* Cleanup files

* Fix metrics setup

* Rename collate function

* Add license to test files

* Rename fixtures

* Add more comments to tiled ensemble training

* Add start of training log message

* Refactor tiler to have explicit arguments

* Refactor pred. storage to have explicit arguments

* Refactor metrics to have explicit arguments

* Refactor visualization to have explicit arguments

* Refactor post-processing to have explicit arguments

* Sort imports

* Add test ensemble script

* Fix join smoothing bug

* Add more documentation to doc-strings

* Remove unused import

* Add brief tiled ensemble documentation

* Update typehints

* Make training args more clear

* Revert addition of no threshold option.

* Refactor normalization and threshold config

* Remove tiled ensemble from docs index

* Add comments to clarify parts of ensemble config

* Improve ensemble config comments

* Add num_tiles attribute to tiler.

* Fix metrics process docstring

* Fix visualization bug and cover with test

* Replace strings with enum

* Improve comments in joiner.

* Fix bug when model doesn't have anomaly maps.

* Improve docstrings (types, clarify).

* Fix visualization tests

* Fix dict membership checks

* Add saving of ensemble config file

* Update test script args

* Cover test script with tests

* Update export warning

* Fix case when no test or val data

* Improve documentation images

* Add images for documentation

* Add codacy suggestion

* Refactor joiner to single class

* Refactor storage names and config

* Update normalization and threshold stage names

* Add transforms independent input size to models

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Make collate function a datamodule attribute

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor tiled ensemble train into pipeline step

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor tiled ensemble prediction into pipeline step

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor tiled ensemble merging into pipeline step

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor tiled ensemble seam smoothing into pipeline step

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor tiled stats calculation into pipeline step

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix ckpt loading when predicting on test set.

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Add logging and add tqdm to pipeline steps.

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor normalization pipeline step

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor thresholding into new pipeline job

* Fix transforms issue when predicting with dataloader

* Add visualization as new pipeline step

* Add metrics as new pipeline step

* Format the code and address some lint problems

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Add code to skip test if test split is none

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Add accelerator to metrics and smoothing

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Make threshold acq helper function and add to threshold to metrics

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Make a separate test pipeline

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Restructure tiled ensemble files into directories

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Pipeline code cleanup

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

* Remove old tiled ensemble files

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Remove old post processing files

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix sigma value read in smoothing

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update stats calc and normalization

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update args naming convention

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor code for nice config

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update docs structure for new system

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Cleanup train code

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix test script args

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update box merging

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor helper function tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Small changes in helper and engine

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor merging tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor tiling tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor metrics test

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Add support for different threshold methods

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Format tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Change test to predict

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor stats calculation tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor prediction data tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update metrics tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Move metrics tests to components

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor seam smoothing tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor normalization tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Move mock stats to conftest

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix typehints for generator

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Refactor threshold tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Temporarily disable box minmax

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Add tiled ensemble integration test

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix normalization tests and add additional merging test

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Add tile collater tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Change dataset in tests to dummy

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Format and fix linter errors

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Format and some cleanup

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Rename predict to eval

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update docs for refactored version of code

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Cleanup the docs

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Update ensemble engine

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Remove boxes from pipelines and tests

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix TODO comment issue

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix unused model in ens. engine

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Fix path case in test

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Change temporary dir to project_path

Signed-off-by: blaz-r <blaz.rolih@gmail.com>

* Change mvtec to MVTec in test path

Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>

---------

Signed-off-by: blaz-r <blaz.rolih@gmail.com>
Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* 📚 Add training from a checkpoint example (#2389)

* Add training from a checkpoint example

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Replace patchcore example with efficient-ad

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

---------

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Export experiment duration in seconds in CSV. (#2392)

* Export experiment duration in seconds in CSV.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Update CHANGELOG

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Log fit and test durations separately.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

---------

Signed-off-by: Weilin Xu <weilin.xu@intel.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* Make single GPU benchmarking 5x more efficient (#2390)

* Use SerialRunner if only one CUDA device is available.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Resolve PLR6201.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Update CHANGELOG.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Keep the same logging level in benchmarking.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

---------

Signed-off-by: Weilin Xu <weilin.xu@intel.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* 🐞 Fix installation package issues (#2395)

* Update the coverage settings

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Remove VlmAd's relative import

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Revert relative imports

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Add type checking

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

---------

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Export the flattened config in benchmark CSV. (#2391)

* Export the flattened config in benchmark CSV.

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Update CHANGELOG

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

* Reuse the existing flatten_dict().

Signed-off-by: Weilin Xu <weilin.xu@intel.com>

---------

Signed-off-by: Weilin Xu <weilin.xu@intel.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>

* `v1.2.0` Release (#2397)

Prepare v1.2.0 release (#2396)

* Update changelog



* Update the version in __init__



---------

Signed-off-by: Samet Akcay <samet.akcay@intel.com>

* Bump Anomalib version to `2.0.0dev` in `main` (#2402)

Update __init__.py

* 🐞Replace package_available with module_available (#2407)

* fix datumaro config

* fix edge case in benchmarking pipeline

* fix vlm ad

* remove ensemble tiling in preparation of refactor

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Samet Akcay <samet.akcay@intel.com>
Signed-off-by: Blaz Rolih <blaz.rolih@gmail.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com>
Signed-off-by: sahusiddharth <siddharth.sahu@plaksha.edu.in>
Signed-off-by: Bepitic <bepitic@gmail.com>
Signed-off-by: blaz-r <blaz.rolih@gmail.com>
Signed-off-by: Weilin Xu <weilin.xu@intel.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Samet Akcay <samet.akcay@intel.com>
Co-authored-by: Blaž Rolih <61357777+blaz-r@users.noreply.github.com>
Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com>
Co-authored-by: Joao P C Bertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Co-authored-by: Marcus Pertlwieser <116986601+Marcus1506@users.noreply.github.com>
Co-authored-by: Siddharth Sahu <112792547+sahusiddharth@users.noreply.github.com>
Co-authored-by: Paco <bepitic@gmail.com>
Co-authored-by: Weilin Xu <weilin.xu@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants