The main objective of this report is to provide recommendations to the European Commission concerning the funding and the steering of public evaluation campaigns as a tool to foster multimedia search technology transfer at the European level. These recommendations rely on the results of 3 complementary preliminary actions conducted within CHORUS+ coordination action.
1- Report on Multimedia Retrieval evaluation dimensions: Based on the literature and CHORUS+ consortium experience in the evaluation of multimedia search technologies, a report synthesizing the role of evaluation at different levels of the innovation workflow was published in November 2011 (deliverable D3.3). International benchmarks and evaluation campaigns were identified as a key component that might fill the gap between scientific criteria used by the researchers (to measure fundamental progress) and industrial & business-oriented criteria used by companies to identify relevant technologies and build innovative products.
2- Survey on the evaluation of Multimedia Search technologies: To better understand the suitability of benchmarking as a tool to foster exchange between academia and industry, a survey was set up and delivered to both communities. The content of this survey was already published within D3.3 deliverable (November 2011). It was subsequently distributed in December 2011 to several communities including ImageCLEF, TRECVID and MIREX benchmarking campaigns, ACM Multimedia 2011 conference, EU projects of the Media Search cluster and a LinkedIn group on enterprise search. The 20 participants to the 5th Think-Tank on Multimedia search technology transfer driven by benchmarking were finally asked to fill the survey. The result’s analysis of the survey was used as a support of discussion during the Think-Tank and is provided in the annex of this report.
3- Think-Tank on Multimedia Search technology transfer driven by benchmarking: A Think-Tank was organized by CHORUS+ on April 19th 2012 during the international conference WWW 2012 in Lyon. This event brought together experts and stakeholders of multimedia search related benchmarking efforts in order to exchange on lessons learned and to assess suitability of benchmarking to foster technology transfer. The results of the above mentioned survey were presented and discussed during the meeting. The question of whether EU should play a role in further strengthening and supporting these efforts was also addressed as an important issue. Around 20 people including leading industrials, expert SMEs, EC representatives and highly known researchers in the multimedia search technology field gathered in Lyon to discuss these subjects. The notes synthesizing the whole discussions of this Think-Tank are available
The content of this report is organized in two main sections. The first part (section 1) synthesized the main conclusions and lessons learned from the three actions described above. The second part provides the recommendations of CHORUS+ consortium towards sustaining and/or improving European practices concerning public benchmarking.
See more
I. Main Conclusions and lessons learned
II. Recommendations
III. Annex 1
- Questionnaire
This report has been prepared under main supervision of Alexis JOLY, INRIA, France
|
Print
