[sigcomm] Workshop on Experimental computer Science

[Dror Feitelson]

Workshop on EXPERIMENTAL COMPUTER SCIENCE
Part of ACM FCRC, San Diego, 13-14 June 2007
(in cooperation with ACM SIGCOMM)

URL: http://www.expcs.org


Call for Papers

This is not a conventional "systems" conference in computer science!
First, it takes a broad view including both software and hardware
systems, networking, applications, software engineering, and even theory.
Second, the focus is on the experimental approach to doing research.
High-quality papers are solicited on the
nine themes identified below. In particular, we proactively encourage
papers that advance the methodological aspects of experimental
computer science, or present new real-world data and observations
about computer systems and use. A major criterion for acceptance will be that
the paper contributes to the discourse on the subject; it does not
have to be the definitive final word on it.

* Experimental engineering:
 techniques, insights, and understanding that come from building and
 using computer systems, and discovering behavior that emerges from
 the inherent complexity of working systems.

* Measurements and metrics:
 what to measure, how to achieve reliable measurements in face of
 noise and errors, and what metrics will turn raw
 measurements into useful information.

* Experimental methodology:
 best practices that should be followed, and pitfalls that need to
 be avoided.

* Tools of the trade:
 measurement tools that support sound instrumentation, tools for
 automatic flagging of anomalous conditions, and tools for automatic
 management of experiments.

* Explanation and hypothesis testing:
 the study of complex systems, with the goal of achieving
 a deeper understanding of why they behave in the way they do.

* Reproducibility:
 reproducing previous results, the scope to which they pertain, and
 factors that affect them.

* Data collection and data manipulation:
 reports on collecting and using data, including issues like
 anonymization and sanitization.

* Theory and practice:
 cases where theory leads to hypotheses that can be checked
 experimentally, and cases where experimentation questions
 assumptions used as a basis for theory.

* Education and an experimental culture:
 the need for a cultural change in order to make the experimental
 approach more prevalent, and development of courses and curricular material.

In the interest of reproducibility and advancing the state of the art,
it is highly desirable that papers be accompanied by software and data
sets used in the experiments.

Measurement Challenge

In addition to contributed papers, we are planning to have a session
devoted to a common measurement challenge, where participants present
(and discuss) how they approached a given problem.
The challenge this year is to measure the indirect overhead of a
context switch, i.e. the degradation in performance due to
perturbation of cache state.
This is non-trivial, as it obviously this depends on myriad details
including workload behavior and architecture.
To participate, submit a report of your approach and results by the
paper submittal deadline.
Selected reports will be included in the proceedings.

Roundtable on Experimentation in Computer Science Education

Experimentation and empirical data seem to be all but absent from
current computer science curricula.
We are planning to hold a roundtable on this issue, with emphasis on
operational ways to advance experimentation in computer science
education, including the design of complete courses, how to add
experimentation into existing courses, and ideas for term projects.

To participate, submit a position paper by the paper submittal deadline.
A joint paper summarizing the roundtable will be included in the proceedings.


For more details, submittals, and up-to-date information,
see http://www.expcs.org