Questionnaire

PART1 of the questionnaire

Using benchmarking for technology

transfer

Q1.1 Which aspects are likely to contribute to the commercial success of a technical component?

Overall results

Detailed results

The results show that according to all groups of respondents, technical performances is the most important aspect that contributes to the commercial success of a technical component. Ease of integration is an important aspect for small companies and developers in general.

Interestingly commercial terms appear to be the less contributive aspect for big companies whereas reciprocally, scientific excellence is considered as the less contributive aspect by academics.

Scientific excellence is overall the less contributive factor to commercial success.

Q1.2 How do you identify new technical components that you would like to experiment and/or benchmark?

Overall results

Detailed results

The results show that according to all groups of respondents, scientific articles are the best way to identify new technologies, followed by benchmarking campaigns.

Interestingly, watching competitors is a good source of information according to big companies whereas small companies and academics ranked it the worst criterion.

Q1.3 What criteria do you use for selecting technical components (for experimentation, proof of concept or integration in products)?

Overall results

Detailed results

Overall results show that benchmarking campaigns are on average the best criteria to select new technical components for integration or deeper testing (whereas in previous question scientific articles were judged as the best way to identify/discover new components). But it is important to notice that benchmarking campaigns are ranked as the second best criteria by almost all groups when looking at the detailed result’s table. Top-1 criteria for academics is actually scientific impact whereas the top-1 criteria for companies is technical skills (e.g. scalability, response times, portability, etc.). In between, benchmarking appears as the best compromise between research (technology suppliers) and exploitation (technology integrators).

PART 1 Synthesis

- Scientific literature is the best way to prospect and discover new technologies

- Technical performances are the best key of commercial success whereas scientific excellence is judged as the worst one

- Academics & Companies differ on how they select technologies in practice (for integration or testing): scholarly impact vs. technical skills

- But Academics & Companies agree on that Benchmarking is a good way to select technologies in practice. So that benchmarking appears as the best compromise between research and exploitation. This central position makes it a powerful tool for boosting technology transfer.

PART2 of the questionnaire

Public Evaluation campaigns

Q2.1 Which evaluation campaign is the most suitable for your business or research activity?

For both academics and companies, TRECVID is far away the most suited evaluation campaigns for their business or research activity, followed by ImageCLEF, MIREX and MediaEval. Notice that these results might be biased by the proportion of respondent’s coming from the TRECVID community. But still, according to other statistics on the number of participants to these different campaigns (provided in D3.3) it is highly believable that TRECVID is the most popular one. In 2011 for instance, the number of participants was 73 at TRECVID, 43 at ImageCLEF, 40 at MIREX, 39 at MediaEval, 25 at PASCAL VOC, 15 at SHREC.
Q2.2 To your opinion, the challenges measured in public evaluation campaigns are

Q2.3 To your opinion, the evaluation criteria used in public evaluation campaigns are

Both academics and companies consider that challenges measured in public evaluation campaign as well the used evaluation criteria are reasonably relevant and very relevant for about 20% of them.

Q2.4 In the future do you plan to

An important conclusion of this graphic is that almost 60% of the companies who responded to the questionnaire plan to use technologies selected as the best ones within benchmarking campaigns.

Future intentions of respondents about their participation in benchmarking campaigns show a stable interest compared to actual participation (45% of respondent’s companies and 70% of respondent’s academics did participate in a campaign in a the past).

PART 2 Synthesis

- There is an agreement on the relevance of existing benchmarks

- 60% of companies plan to use technologies selected by benchmarks

- Attractiveness to participate in and organize public benchmarks is still there

PART3 of the questionnaire

Scientific Evaluation Criteria

Q3.1 What are the best criteria that you think should be taken into account when benchmarking multimedia IR components?

Overall results

Detailed results

User satisfaction criteria alone

The results show that according to all groups of respondents effectiveness is the best criteria to evaluate multimedia IR components, followed by scalability and efficiency. Ergonomy of the Graphical User Interface is not considered an important criterion in such evaluations.  Looking at the user satisfaction criterion alone, we see that a majority of big companies would like to see some user trials in benchmarking campaigns.

Q3.2 What criteria do you use to judge that a scientific article is an important contribution?

Overall results

Detailed results

Scientific excellence and biblio-metrics (number of citations, H-index, etc.) are ranked first by managers and big companies. On the other side, experimental results are ranked first by small companies, developers and academics. Finally, claims of the authors of a paper are ranked first by students whereas they are among the worst criterion for big companies and managers.

Overall, we can remark that:

- Confidence in claims decreases with financial impact of the respondents

- Confidence in research community increases with financial impact of the respondents

- Relevance of experimental results is quite stable over the different groups and on the average the best criterion

Q3.3 What are the greatest difficulties in the scientific evaluation of multimedia retrieval?

Overall results

Detailed results

The results clearly show that according to all groups of respondents data availability is most critical issue in evaluating multimedia retrieval technologies. Human resources appear as the second main limitation whereas hardware resources do not appear as a problem. This last point has to be mitigated by the fact that the scale of currently available data is relatively small compared to real-world data. So that if the main limitation (data availability) was solved, it is probable that hardware limitations would become more critical to process (to process very large amount of data).

PART 3 Synthesis

- Effectiveness is considered as the top-1 evaluation criterion and this validates the approach of current benchmarking campaigns. It is followed by scalability and Efficiency concerns. Only big companies are convinced by human-centered evaluation as a complementary criterion to be used in benchmarking campaigns.

- Criteria used to evaluate scientific publications are diverse and evolve with the financial impact of the underlying decisions to be taken. Experimental results are the most consensual criteria.

- Data availability is most critical issue in evaluating multimedia retrieval technologies