Check out our manually-crafted benchmark ClassEval for evaluating LLMs on class-level code generation. Any feedback would be appreciated for help us improve the benchmark. GitHub, Preprint