The objective of this work package is to define a benchmarking methodology and scenarios for networking experiments in specific environments (peer-to-peer and wireless networks) and to provide an experimental workflow for networking experiments. In OneLab2, WP9 will focus on the development of the basic concepts and methods, while in a future phase, WP9 will move towards a (partial) automation of the benchmarking process.

Evaluation of new network protocols and architectures are at the core of networking research. This evaluation is usually performed using simulations (e.g., NS2), emulations (e.g., Emulab), or in experimental platforms (e.g., PlanetLab). Simulations allow a fast evaluation process, fully controlled scenarios, and reproducibility. However, they lack realism and the accuracy of the models implemented in the simulators is hard to assess. Emulation allows controlled environment and reproducibility, but it also suffers from a lack of realism. Experimentations allow more realistic environment and implementations, but they lack reproducibility and ease of use.

Therefore, each evaluation technique has strengths and weaknesses. However, there is currently no way to combine them in a scientific experimental workflow. Typical evaluation workflows are split into four steps: topology description and construction, traffic pattern description and injection, trace instrumentation description and configuration, and analysis based on the result of the trace events and the status of the environment during the experimentation. To achieve the integration of experimental workflows among the various evaluation platforms, the two following requirements must be verified:

• Reproducibility
  A common interface for each platform must be defined so that the same script can be run transparently on different platforms. This also implies a standard way to describe scenarios including: the research objective of the scenario, topology description and construction, the description of the traffic pattern and how it is injected into the scenario, the description and configuration of the instrumentation and the evolution of the environment during the experimentation.
• Comparability
  As each platform has different limitations, a way to compare the conclusions extracted from experiments run on different platforms, or on the same platform but with different conditions (this is in particular the case for in the wild experimental platforms) must be provided.

Benchmarking is therefore the function that provides a method of comparing the performance of various subsystems across different environments. Both reproducibility and comparability are essential to benchmarking. A networking experiment consists in running a network application (e.g., BitTorrent) in a given environment (wireless mesh network). The environment is described by a set of parameters representing (e.g., the topology, the background traffic, etc). The application performance is measured through a set of relevant metrics (e.g., the mean download time, application data units' loss rate, etc). The results of an experiment are therefore a set of metric/parameter curves.

In a real network overlay like PlanetLab Europe, the environment is not controllable. It is however important to compare the performance of a given application in different environments. Identifying good metrics and parameters and studying the impact of parameters on metrics help to perform the comparison. In WP9, we will run extensive experiments in different environments (in particular wireless networking environments), and study the impact of environment parameters on the performance metrics. Statistical analysis will help identifying parameters that have impact and should not be ignored. We will restrict the study in this project to one specific application (peer-to-peer file replication). As for the environments: we will consider carefully both wireless and mobile environments. However, it is important to stress that we will consider "basic" scenarios with the objective of providing a proof of concept of the benchmarking methodology.

In the framework of this project we will provide a basic benchmarking methodology and scenarios a prototype implementation of a common programming interface between a simulator (NS3) and PlanetLab Europe. This prototype version will allow to run the basic benchmarking scenarios on both platforms (NS3 and PlanetLab Europe) and to compare the experiments results according to the benchmarking methodology defined in the WP. The long-term objective (outside the scope of OneLab2) is to design generic benchmarking methodology and scenarios, and to develop the corresponding validation environment.

Relevant deliverable:

• D9.1 Benchmarking methodology and metrics