Skip to content

fix: use shallow clone to prevent reward hacking via git history #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
neubig wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: use shallow clone to prevent reward hacking via git history #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
157a4ff
fix: use shallow clone to prevent reward hacking via git history
openhands-agent Feb 17, 2026
07ad9d3
fix: fetch base_commit in shallow clone to enable git reset
openhands-agent Feb 18, 2026
118bf36
ci: use local Docker backend instead of Modal for CI tests
openhands-agent Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 4 additions & 10 deletions .github/workflows/system.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,19 +25,13 @@ jobs:
- name: Get tests
run: uv run commit0 get-tests simpy
- name: Test
env:
MODAL_TOKEN_ID: ${{secrets.MODAL_TOKEN_ID}}
MODAL_TOKEN_SECRET: ${{secrets.MODAL_TOKEN_SECRET}}
run: |
uv run commit0 test simpy tests/test_event.py::test_succeed --reference --rebuild
uv run commit0 test simpy tests/test_event.py::test_succeed --reference
uv run commit0 test simpy tests/test_event.py::test_succeed --reference --rebuild --backend local
uv run commit0 test simpy tests/test_event.py::test_succeed --reference --backend local
- name: Evaluate
env:
MODAL_TOKEN_ID: ${{secrets.MODAL_TOKEN_ID}}
MODAL_TOKEN_SECRET: ${{secrets.MODAL_TOKEN_SECRET}}
run: |
uv run commit0 evaluate --reference --rebuild
uv run commit0 evaluate --reference
uv run commit0 evaluate --reference --rebuild --backend local
uv run commit0 evaluate --reference --backend local
- name: Lint
run: uv run commit0 lint commit0/harness/
- name: Save
Expand Down
13 changes: 11 additions & 2 deletions commit0/harness/spec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,13 @@ def make_repo_script_list(self) -> list[str]:
base_commit = self.instance["base_commit"]

setup_commands = [
f"git clone -o origin https://github.com/{repo} {self.repo_directory}",
# Use --depth 1 for shallow clone to prevent agents from accessing
# git history and exploiting it to retrieve original implementations
f"git clone --depth 1 -o origin https://github.com/{repo} {self.repo_directory}",
f"chmod -R 777 {self.repo_directory}", # So nonroot user can run tests
f"cd {self.repo_directory}",
# Fetch both commits needed: env_setup_commit for setup and base_commit for later reset
f"git fetch --depth 1 origin {env_setup_commit} {base_commit}",
f"git reset --hard {env_setup_commit}",
# Remove the remote so the agent won't see newer commits.
"git remote remove origin",
Expand Down Expand Up @@ -217,10 +221,15 @@ def make_repo_script_list(self) -> list[str]:
if version < 7:
specs["python"] = 3.7

base_commit = self.instance["base_commit"]
setup_commands = [
f"git clone -o origin https://github.com/{repo} {self.repo_directory}",
# Use --depth 1 for shallow clone to prevent agents from accessing
# git history and exploiting it to retrieve original implementations
f"git clone --depth 1 -o origin https://github.com/{repo} {self.repo_directory}",
f"chmod -R 777 {self.repo_directory}", # So nonroot user can run tests
f"cd {self.repo_directory}",
# Fetch base_commit needed for eval script reset
f"git fetch --depth 1 origin {base_commit}",
# Remove the remote so the agent won't see newer commits.
"git remote remove origin",
f"uv venv --python {specs['python']}",
Expand Down