China's logo










The CHINA Benchmark Suite

[Page last updated on 2001/02/19 10:29:56.]


The benchmark suite we use for the development and tuning of the CHINA analyzer is probably the largest one ever employed for this kind of purposes. The suite comprises all the Prolog and CLP programs we have access to (i.e., everything we could find by systematically dredging the Internet), for a total of 11 MB of code, 400.000 lines, the largest program containing 10063 clauses in 45658 lines of code.

You can get an idea of what constitutes the benchmark suite by looking at some numbers about the Prolog programs. Note that this table is seldomly updated and it does not contain the synthetic benchmarks (beasts that we generate on a variety of sizes and that are very useful for test coverage and regression testing). Another benchmark not mentioned in the table is a database derived from the first published edition of the Collins English Dictionary. It consists of 40 MB of code for a total of 788708 lines, and it has been quite useful to ensure that CHINA fulfills the ``never crash'' motto.

I am slowly going back to analyze each program in the suite to reconstruct his hystory and give appropriate credit. You can look at the partial program list.

If you have questions on the benchmark suite, please try browsing the page on questions and answers.

Much More Is Needed

Said that, we cannot fail to mention that a much bigger suite would be needed for really serious experimental work. As part of the current development effort, we need tons of real code. Any dialect of Prolog and any CLP language is fine. The programs we will collect will be used for performance evaluation, regression testing, and gathering of statistical data.

Notice that there is no restriction whatsoever: programs can be in any (C)LP language, they are not required to work in any intended way, they can be incomplete (e.g., isolated modules or libraries), they can use any kind of non-logical feature, they can refer to foreign predicates (e.g., written in C). Good style or documentation is not required, even though the availability of some documentation is a bonus. For instance, if you have conserved the Prolog assignments done by your students, it is not necessary that you lose your time: do not even look to them, just pack everything (tar, zip, or anything else) and send the result. Sending an URL is fine, of course. Big programs and small programs are equally appreciated. Also, if you know of any logic programming code repository or package, please let us know.

Please, send any contribution to and thanks for your cooperation.

© The China
Development Group

Home | Systems | Documentation | Experiments | Benchmarks | Projects | People | Links | About