add SWE-bench dev set support #3477

[jatinganhotra]

What problem or use case are you trying to solve?
Linked to feat: add SWE-bench fullset support #3477
#3477 adds instance level images for test sets. It would be awesome if you could add 'princeton-nlp/SWE-bench', split='dev' and 'princeton-nlp/SWE-bench_Lite', split='dev') as well.

Describe the UX of the solution you'd like

Do you have thoughts on the technical implementation?

Describe alternatives you've considered

Additional context

[jatinganhotra]

Copying the link to discussion from #3477 here: https://github.com/All-Hands-AI/OpenHands/pull/3477/files/a8e35be0b7f95b9018c6d0976615397500f287a1#r1741178721 as it was closed and merged.

@xingyaoww we're still waiting for SWE-Bench authors to resolve an issue, which is blocking adding 'dev' set evaluation. I reached out to them and will share here when they have resolved it.

[jatinganhotra]

Hi @xingyaoww , the blocker issue princeton-nlp/SWE-bench#199 on the SWE-bench has been fixed now.

So, it would be great if we can revive this issue when you get a chance.

It'd be great if you can share how you create instance-level images. I couldn't find the scripts/ README to create instance-level images. I can try and help out by creating a PR if you'd like.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for over 30 days with no activity.