Performance Indicators

Contact persons for Performance Indicators: Victoria Henson-Apollonio.

Contributors to this page: CGIAR CAS-IP, Rome, Italy (Victoria Henson-Apollonio); Global Crop Diversity Trust, Rome, Italy (Mellissa Wood, Jane Toll, Hang Nguyen); Bioversity International, Rome, Italy (Brigitte Laliberte, Jonathan Robinson, Jamie Watts); IRRI, Los Baños, Philippines (Ruaraidh Sackville Hamilton); IITA, Nigeria (Dominique Dumet); CIMMYT, Mexico (Thomas Payne).

Introduction

Performance Indicators (PIs) are a way to attempt to succinctly measure progress made (over time) towards achieving a goal. Ideally, PIs are quantifiable rather than qualitative, allow an analysis of trends, and are agreed upon by the organization or group that is using them. Obviously, goals change and PIs need to be reviewed on a periodic basis. PIs should help an organization improve its performance in achieving an agreed set of goals.

The activities that have led to a set of PIs for the CGIAR genebanks included:

• A background paper that describes the current state of discussion regarding the usefulness of indicators in general, the potential effects of using PIs to measure the performance of genebanks, and the indicators that had been developed by other genebanks.
• A meeting of the activity coordinators and interested parties for several of the Global Public Goods (GPG) activities related to PIs such as the “Best Practices”, “Decision Support Tools for Financing”, “Risk Management Guidelines”, and “Identification of User Groups”.
• A workshop that developed the initial set of GPG2-PIs, that included participation by the CGIAR genebank managers, managers of other public genebanks that had developed PIs and performance measurement systems, the Crop Diversity Trust, and professionals that develop and maintain quality management systems (QMS). Recognising the importance of not placing an unnecessary measurement and reporting burden onto already strapped genebanks, it seemed logical and efficient for the Trust to contribute to the development of these indicators with genebank managers, rather than develop an additional set in isolation, to do essentially the same thing.
• Further development and utilization of the initial set of PIs by the Trust for their long-term grant technical reporting.
• Incorporation of goals that were described in the GPG2-developed sustainability plan and human resources plans.
• Adjustment review of the PIs by the Trust based on feedback from reports from genebanks working with the Trust.
• Adjustment of the PIs, based on feedback from three CGIAR test genebanks.
• Final activity discussions with the Trust to produce a harmonized set of PIs.

In essence, two parallel streams of PI development activity took place over a three-year period. The Trust, building on the work resulting from the Lunteren workshop, further developed a set of PIs for assessing performance (primarily requiring quantitative data) that were utilized by partner genebanks over a two-year period. There was some review after the first year. In parallel, GPG2 further developed a set of PIs (more focused on processes) that were tested by three genebanks. A number of times these PI activities came together for alignment where possible (recommending that the same genebanks be involved in the reporting). Finally the genebanks involved requested only one set of PIs be used, recommending the Trust set as the base as they had all actively been involved in its development and testing. As a result, the two streams were completely harmonized in January 2010.

Recommended joint indicators

Technical indicators

Global system genebank indicators

References and further reading

Batterham R. 1995. How can we argue with performance indicators? In: Problems with performance indicators. Centre for Health Program Evaluation. Working Paper 54.

Campbell DT. 1979. Assessing the impact of planned social change. Evaluation and Program Planning, 2:67-90

CGIAR. 2006. Results 2005 for the CGIAR Performance Measurement Excercise. [online] Available from: http://www.cgiar.org/pdf/agm06_pm_results_2005.pdf. Date accessed 27 April 2010.

Department for Environment, Food and Rural Affairs. 2006. Environmental key performance indicators. Reporting guidelines for UK businesses, 76pp.

de Vincente MC. 2004. The evolving role of genebanks in the fast-developing field of molecular genetics. Issues in Genetic Resources. No. 11, International Plant Genetic Resources Institute, Rome, Italy.

European Academies' Science Advisory Council. 2004.  A user's guide to biodiversity indicators. 60pp.

FAO/IPGRI. 1994. Genebank standards. Food and Agriculture Organization of the United Nations, Rome and International Plant Genetic Resources Institute, Rome. Available in English, Spanish, French and Arabic.

Gawande A. 2006. The Score, The New Yorker. [online] Available from: http://www.newyorker.com/printables/fact/061009a_fact. Date accessed 27 April 2010.

Grol R. 2000. Between evidence-based practice and total quality management: The implementation of cost-effective care. International Journal for Quality in Health Care, 12:297-304.

Guidelines for the reporting of performance indictors for CGIAR Centers. 2006 data. Science Council and CGIAR Secrtariat, January 11, 2007.

Houghton G, Rouse A. 2004. Are NHS primary care performance indicators scores acceptable as markers of general practitioner quality? British Journal of General Practice, May 2004:341-344.

Ibrahim JE. 2001. Performance indicators from all perspecives. International Journal for Quality in Health Care, 13:431-432.

Jones PD, James N, Mellalieu SD. 2004. Possession as a performance indicator in soccer. International Journal of Performance Analysis in Sport, 4:98-102.

Lindal M. 2000. Forest sector indicators, An approach for Central America. CIAT, World Bank, UNEP, 48 pp.

Millennium development holes. 2007. Nature, 446:347.

Perrin B. 2002. How to - and how not to - evaluate innovation. Evaluation, 8:13-28.

Rao NK. 2006. Managing seed genebanks - Challenges and options for improving efficiency. IPGRI. Newsletter, January 2006:5-6.

Rao NK, Hanson J, Dulloo ME, Ghosh K, Nowel D, Larinde M. 2006. Manual of seed handling in genebanks. Handbooks for Genebanks No. 8. Bioversity International, Rome, Italy. Available in English (1.5 MB),  Spanish (1.4 MB) and French (1.9 MB).

SINGER. (Homepage of SINGER) [online] Available from: http://www.singer.cgiar.org. Date accessed 27 April 2010.

Van Koppen B. 2002. A gender performance indicator for irrigation: Concepts, tools and applications. IWMI Research Report no. 59, 52pp.

Van Thiel S, Leeuw FL. 2002. The performance paradox. Public Performance and Management Review, 25:267-281.

Quality management

Contact person for Quality management: Jean Hanson, ILRI, Ethiopia

Contributors to this page: Bioversity International/ILRI, Ethiopia (Alexandra Jorge); CSL (David Galsworthy); CIP (Edwin Rojas).

This page provides the latest updates on Quality Management Systems (QMS) for genebanks.

There is increasing pressure to define and use criteria of quality for commercial as well as research processes and procedures. The underlying concepts have been applied in many areas of modern society to control risk and provide the mechanism for quality improvement. Manufacturing, food production, pharmaceutical industry, environmental controls and information technology are just some of the areas that rely on the application of quality standards and systems to support business. However, the biological/agricultural sciences have been slow to recognize the benefits of these systems and there have been limited Quality Management System (QMS) application in these fields.

A QMS can be defined as a set of policies, processes and procedures required to plan and execute the activities of any genebank. Implementing a QMS within any organization needs to be a top management decision.

An adequate and suitable QMS should have the capacity for improving the quality of operations by:

• Demonstrating control of critical risks.
• Improving control of proper functioning of equipment and quality of supplies.
• Improving quality control of processes.
• Formalizing training of staff and the demonstration of staff competence.
• Supporting best practice implementation.
• Compliance with any audit procedures.

The implementation of an appropriate QMS is particularly relevant for genebanks which are constantly striving for the highest standards in order to conserve and maintain crop genetic resources. Key genebanks have recently taken the lead in the implementation of QMS using the framework of the International Organization for Standardization (ISO) quality standards and have gained certification (ISO 9001) or accreditation (ISO 17025) of their operations. This has allowed an analysis of the possibility of using ISO quality standards more widely and broadly within its genebank community.

Options for QMS of genebanks

Options ranging from a minimum documentation for a quality system through to formal recognition through accreditation are all possible levels of implementation of a QMS suitable for genebank management:

1. Certification – Product certification refers to the processes intended to determine if a product meets minimum standards, similar to quality assurance, which is usually related to administrative operational procedures. The ISO 9001 is a generic management standard that can be applied to any business enterprise, public administration, government department or research institute. It has been successfully used in laboratories and genebanks. Certification to ISO 9001 standard does not guarantee the compliance (and therefore the quality) of any end product or service; rather, it certifies that consistent processes are used and the system of an organization is being applied to assure consistency. A pre-requisite for an organization to become certified is to have a documented quality management system that follows the content of ISO 9001. Examples of genebanks using certification are the Centre for Genetic Resources, the Netherlands and Institute of Plant Genetics and Crop Plant Research, Germany.

2. Accreditation – The accreditation process ensures that certification practices of competency, authority or credibility are acceptable, typically meaning that they are competent to test and certify third parties, behave ethically and employ suitable quality assurance. It is usually related to technical operational procedures. The ISO 17025 is aimed at improving an organization’s ability to consistently produce valid results and it has been successfully used for laboratories and also genebanks. ISO 17025 comprises both management and technical requirements. The management requirements are primarily related to the operation and effectiveness of the quality management system. The technical requirements address the competence of staff, validation of methodology and equipment control. Since this standard is about competence, accreditation is used for the formal recognition of that competence. In the same way as ISO 9001, a pre-requisite for an organization gaining accreditation is to have a documented quality management system that follows the requirements of ISO 17025 standard. This quality standard allows organizations to carry out procedures in their own way, but requires the demonstration of continuous improvement, by regular audit. An example of a genebank using accreditation is the International Potato Centre (CIP), Peru.

3. Documentation of Processes - Each genebank should define, document and adopt the processes employed to reach the appropriate minimum standard for routine procedures, taking into account specific risks and local conditions. Regular revisions and necessary updates may be necessary to this minimum specification. Documentation of processes is vital to enable the capture of knowledge accumulated by staff over many years. Lack of proper recording and documentation with limited accessibility to the information is a major issue. The completion of detailed documentation for genebank operations is seen as a minimum requirement for high quality genebank management and could eventually lead to full implementation and more formal verification of performance through certification and accreditation at a time when it is felt that the genebank is ready for moving to this level. Many genebanks are already using this approach.

4. Designer solution – A combination of QMS approaches can also be a practical solution for genebanks. ISO certification or accreditation can be used for particularly critical or high risk activities/procedures of the genebank together with documentation of processes for routine less risky operations. This compromise would guarantee a more controlled and reliable approach to key procedures and selected genebank activities. An example of a genebank using the designer approach is CIMMYT, where the genebank operations follow documented procedures but the seed health laboratory uses ISO standards.

Whichever option is selected the best value is obtained by balancing the use of resources on the one hand and perfection on the other. The objective of the quality system needs to be clearly defined so that the system can be effective. The design and implementation of a quality management system will vary depending on the type, size and products of the genebank. While there are many similarities between genebanks, allowing systems already created for some genebanks to be partially adapted/adopted by other similar genebanks, each genebank will have its own procedures depending on crop and management.

The comparison of options concluded that:

The highest level of quality assurance, as well as the most prestigious, is achieved with accreditation but this comes together with a higher financial cost and commitment.

An intermediate and possibly equally good option is the designer option, with careful choices of critical areas.

The least costly solution appears to be the Documented Procedures. This has the possibility to be upgraded into any of the other full certification and accreditation ISO solutions at a later stage, allowing for a gradual change.

Implementation of a QMS of genebanks

Steps in the selection and implementation process for a QMS are:

• Evaluate the external factors.
  • Seek information regarding external policies and requirements that will impact on genebank procedures.
• Evaluate the internal factors.
  • Assess the present system and resources available for implementation.
• Evaluate QMS model to apply.
  • Gather information and determine the complexity and the amount of work required for different models and how this will be applied for the crop.
• Decide on QMS options.
  • Based on the above information, determine which model best fits the needs.
• Formulate an implementation plan.
  • The overall implementation plan should include: objectives, milestones and responsibilities for all the key areas of a QMS:
    • Overall system design.
    • Quality procedure production.
    • Documentation of procedures and workflows (ideally this should be already in place in any genebank system).
    • Introduce a record system to allow re-creation of all activities at a later stage.
    • Internal audit training.
    • Internal audit plan and implementation.
    • Production of training records.
    • Establish environmental controls as necessary.
    • Validate methodology of processes.
    • Produce equipment records and evidence of equipment control.
    • Produce evidence of compliance.
• Implementation
  • Implementation should be regularly monitored to ensure milestones are appropriate and met.
• Independent verification if desired/required
  • This is only applicable for the certification to ISO 9001 or accreditation to ISO 17025. The normal process would involve the initial dialogue with the certification/accreditation body, pre-assessment visit, then assessment visit and finally the award of certification/accreditation.
• Ensure maintenance of the system
  • The system needs to be regularly reviewed to ensure it is being adequately maintained. This includes regular surveillance visits in the case of certification or accreditation.

Recommended best practices for QMS of genebanks

The main recommendations for application of a QMS were:

• Any genebank should have a good a reliable QMS in place, properly documenting all the procedures, as a minimal baseline.
• Accreditation would be the best technical option to minimize the risk of loss of diversity in crop genebanks. If resources allow, the implementation of ISO 17025 is highly recommended but ISO 9001 is a good option.
• The designer solution provides flexibility using risk/vulnerability based choices to determine which areas require accreditation.
• Certification or Documented Procedures alone are not the best technical choices for genebanks with critical and risky areas.

Relevant adequate certification/accreditation bodies should be approached to consider a formal quality assurance ISO for genebanks.
Click on ISO for genebanks to read the full document about the feasibility study and major recommendations that were submitted to the SGRP in December 2008. Click also on the table of addressed changes to read about the major comments, suggestions and changes made.

You can also check the latest updates from AEGIS regarding regional network approaches.

References and further reading

A European Genebank Integrated System. [online] Available from: http://www.ecpgr.cgiar.org/AEGIS/AEGIS_home.htm/. Date accessed: 08 April 2010.

ACI Global. 2010. Overview of ISO 9001:2008. ACI Global Pty Ltd. [online] Available from: http://www.aciglobal.com.au/ISO9001_Overview.html. Date accessed: 08 April 2010.

American Association for Laboratory Accreditation. [online] Available from: http://www.a2la.org/. Date accessed: 08 April 2010.

American Society for Quality. [online] Available from: http://www.asq.org/. Date accessed: 08 April 2010.

Anon. 2002. Feedback on ISO 9001:2000. ISO Management Systems, Special Report. pp 19-29 [online] Available from: http://www.bin.co.uk/info/9k2kfeedback.pdf. Date accessed: 08 April 2010.

Anon. 2002. Reactions to ISO secretary-general’s ‘police yourselves’ call to ISO 9000 community. ISO Management Systems, Debate pp 55-61. [online] Available from: http://www.bin.co.uk/info/Reactions_IMS.pdf. Date accessed: 08 April 2010.

Centro de estudos da Macaronésia (CEM). O Centro: ISOPlexis. [online] Available in Portuguese from: http://www.uma.pt/Investigacao/Ccbg/swebs/germoplasma/index.html. Date accessed: 26 March 2010.

CGN quality management system. [online]. Center for Genetic Resources, the Netherlands. Available from: http://www.cgn.wur.nl/UK/CGN+General+Information/Quality+manual/ Date accessed: 10 February 2011.

Chartered Quality Institute (CQI) . [online] Available from URL: http://www.thecqi.org/ Date accessed: 08 April 2010.

Engels J, Maggioni M. 2008. Quality Management System for AEGIS. A European Genebank Integrated System. Meeting in Poland. Bioversity International, Rome, Italy. [online] Available from: http://www.ecpgr.cgiar.org/AEGIS/Meeting_Poland0708/QMS_draft.pdf. Date accessed: 08 April 2010.

European cooperation for Accreditation (EA). [online] Available from: http://www.european-accreditation.org/default_flash.htm. Date accessed: 08 April 2010.

Honsa and McIntyre. 2003. ISO 17025: Practical benefits of implementing a quality system.

International Food Safety & Quality Network. Food Safety Standards Directory. [online] Available from: http://ifsqn.com/links.php. Date accessed: 08 April 2010.

International Organization for Standardization (ISO). 2010. ISO forms, model agendas, standard letters. [online] Available from: http://www.iso.org/forms. Date accessed: 08 APril 2010.

International Organization for Standardization (ISO). 2010. ISO/IEC directives and ISO supplement. [online] Available from: http://www.iso.org/directives. Date accessed: 08 April 2010.

International Organization for Standardization (ISO). 2010. ISO templates. [online] Available from: http://www.iso.org/templates. Date accessed: 08 April 2010.

International Organization for Standardization (ISO). 2007. Complete collection of generic ISO 9000 standards. [online] Available from: http://www.iso.org/iso/pressrelease.htm?refid=Ref1091. Date accessed: 08 April 2010.

ISO 17025 Quality Manual template. [online] Available from: http://www.e-shoq.com/?gclid=CKmq5_jYspUCFQnIsgodYQz_Pw. Date accessed: 08 April 2010.

Quality Network. 2006. ISO 17025 Competence of Testing and Calibration Laboratories. [online] Available from: http://www.quality.co.uk/custpage.htm. Date accessed: 08 April 2010.

Rtech Laboratories. 2003. ISO 17025: Practical Benefits of Implementing a Quality System. Journal of AOAC International. [online], 86 (5), pp.1038-1044. Available from: http://www.atypon-link.com/AOAC/doi/abs/10.5555/jaoi.2003.86.5.1038?journalCode=jaoi. Date accessed: 08 April 2010.

The Food and Environment Research Agency (FERA). [online] Available from: http://www.fera.defra.gov.uk/. Date accessed: 08 April 2010.

The Business Improvement Network. [online] Available from: http://www.bin.co.uk/. Date accessed: 08 April 2010.

United Kingdom Accreditation Service (UKAS). [online] Available from: http://www.ukas.com/. Date accessed: 08 April 2010.

Wade J. 2002. Is ISO 9000 really a standard? ISO Management Systems, Viewpoint. pp.17-20. [online] Available from: http://www.bin.co.uk/info/IMS_May_2002.pdf. Date accessed: 08 April 2010.

Wikipedia. 2010. List of ISO standards. Wikimedia Foundation, Inc. [online] Available from: http://en.wikipedia.org/wiki/List_of_ISO_standards. Date accessed: 08 April 2010.

General genebank management strategies and principles

Please click on one of the menu items on the left to proceed.

This section of the website covers important strategies and principles related to genebank decision-making, that are not necessarily crop-specific.
 
The pages listed on the left show the outputs developed by different groups, leaders of parallel activities of the project on the “Collective Action for the Rehabilitation of Global Public Goods in the CGIAR Genetic Resources System: Phase 2”, funded by the World Bank. 

Some of these outputs are still under preparation, but will be posted as soon as they are released.

The persons responsible for the information provided in these pages are listed at the top of each page.

Genetic identity

Maintenance of the genetic identity of accessions during genebank management is one of the cornerstones of genetic resources conservation. Accessions maintained in genebanks should remain as genetically similar to the original collected material as possible. It is very important to prevent the unintentional introgression of exotic genes, including transgenes, not already present in samples conserved in germplasm collections. Genebank processes are designed to avoid inadvertent introduction of genes through geneflow, avoid selection and retain genes in low frequencies within the population to ensure that maximum diversity is retained for future use. Best practices in genebanks should be able to achieve a high degree of probability that an accession maintains its original genetic identity over generations of regeneration and storage.

Genetic integrity can be at risk at all stages of the genebank handling process and careful handling procedures should be put in place to reduce the risk of loss of alleles or genes within the population or accession. These include paying attention to uneven loss of genotypes during storage by ensuring viability does not decline to low levels, using adequate sample size to capture genes in low frequencies for regeneration, providing isolation from pollen contamination and reducing the regeneration interval and carrying out regeneration in environments where the germplasm is well adapted to minimize selection and genetic drift during regeneration.

Even with careful handling, accessions may face genetic erosion or lose genetic integrity during long term storage over many cycles of regeneration. Routine testing schemes are required to monitor genetic identity and genetic erosion during genebank handling. In the past these tests have relied on checking morphological traits against standard lists of descriptors and comparing the traits in each generation to the original phenotype or population. New molecular tools have now provided the opportunity to monitor genetic integrity at the genotype level and laboratory tests are available to determine any unintentional genetic erosion or change in genetic identity.

This section provides information on the standards used to describe diversity and the methods used to detect changes in genotype, unintentional loss of alleles or genes or unintentional inclusion of genes and transgenes into accessions. Information is available on:

• Genetic integrity analysis.
• Transgene analysis.
• Characterization standards.

References and further reading

Hirano R, Jatoi SA, Kawase M, Kikuchi A, Watanabe N. 2009. Consequences of ex situ Conservation on the Genetic Integrity of Germplasm Held at Different Gene Banks: A Case Study of Bread Wheat Collected in Pakistan. Crop Science 49:2160-2166.