Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full experiment with the new C-specific prompt #370

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

DonggeLiu
Copy link
Collaborator

Running a full experiment with the new C-specific prompt.

@DonggeLiu DonggeLiu added the Experiment-only A PR only to run experiments, do not merge it to main. label Jun 21, 2024
@DonggeLiu
Copy link
Collaborator Author

@DavidKorczynski: Are there any other changes required to use the C-specific prompt?
Would you recommend updating the benchmarks before the exp?

@DonggeLiu
Copy link
Collaborator Author

/gcbrun exp -n dg

@DavidKorczynski
Copy link
Collaborator

DavidKorczynski commented Jun 21, 2024

@DavidKorczynski: Are there any other changes required to use the C-specific prompt?

No

Would you recommend updating the benchmarks before the exp?

Yeah, I don't think it makes too much sense to try on C++ unless we're testing that it doesn't break the workflow or so -- I wouldn't expect the generated prompts to generate good C++ harnesses. We can use these benchmarks for pure C projects only: #371

DonggeLiu pushed a commit that referenced this pull request Jun 21, 2024
Adds a set of benchmarks that have been extracted for the projects 
- clib
- htslib
- croaring
- kamailio
- opensips
- unit
- ntpsec
- bind9
- libyang
- miniz
- mdbtools
- libiec61850

and using the oracles

- far-reach-low-coverage
- low-cov-with-fuzz-keyword
- easy-params-far-reach

Ref: #370

Signed-off-by: David Korczynski <[email protected]>
@DavidKorczynski
Copy link
Collaborator

/gcbrun exp -n dk-9000 -b c-specific

@DonggeLiu
Copy link
Collaborator Author

DonggeLiu commented Jun 24, 2024

The experiment report with c-benchmarks only:
https://llm-exp.oss-fuzz.com/Result-reports/scheduled/2024-06-23-weekly-all/

## The incorrect header error persists.
Take bind9 as an example:
The prompt provided the header full path (/src/bind9/lib/dns/include/dns/view.h), but LLM still used the incorrect path (/src/bind9/lib/isc/include/dns/view.h), which highlights the need to fix this programmatically.
BTW, LLM generated the correct header with the default prompt.

I reckon to avoid regression, we can:

  1. Only fix the header when compilation fails due to this.
  2. Fix header programmatically: 1) Parse the wrong header from the error message, 2) find the correct file path, 3) replace the wrong line with the correct path, 4) compile the fuzz target again.

We are trying a different solution to this.

Missing function implementation

Here is an example.

@DavidKorczynski will you address these?

@DonggeLiu
Copy link
Collaborator Author

DonggeLiu commented Jun 24, 2024

LLM code-bison-32k failed to generate results of some benchmarks because it is not available in certain regions.
Re-running under the same config as 2024-06-23-weekly-all/ with C benchmarks only using Gemini-1.5:
https://llm-exp.oss-fuzz.com/Result-reports/scheduled/2024-06-24-weekly-all/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Experiment-only A PR only to run experiments, do not merge it to main.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants