Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

much baseline diff compared rankllama #2

Open
stanpcf opened this issue Feb 6, 2024 · 1 comment
Open

much baseline diff compared rankllama #2

stanpcf opened this issue Feb 6, 2024 · 1 comment

Comments

@stanpcf
Copy link

stanpcf commented Feb 6, 2024

image

such as BM25'avg in beir should be 43.7

@stanpcf stanpcf changed the title 为什么你们论文里多处数据指标和论文重点对比的rankLLama指标对不上 much baseline diff compared rankllama Feb 6, 2024
@zlh-source
Copy link

zlh-source commented Feb 8, 2024

  1. The Parameter Configurations of the paper RankT5 shows that RankT5 uses T5-large (770M) for initialization.

  2. The results in Tables 3 and 4 in our paper are both ranking the top 1000 documents retrieved by BM25. According to our email communication with the authors of RankLLaMA, the RankLLaMA results in their paper are based on the top 100 documents retrieved by RepLLaMA.

  3. The performance of RankLLaMA in our paper can be reproduced through the official script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants