UNIT 63 - BENCHMARKING

UNIT 63 - BENCHMARKING

  • A. INTRODUCTION
  • B. QUALITATIVE BENCHMARKS
  • C. QUANTITATIVE BENCHMARKS
  • D. EXAMPLE MODEL OF RESOURCE UTILIZATION
  • E. APPLICATION OF MODEL
  • F. LIMITATIONS
  • G. AGT BENCHMARK EXAMPLE
  • REFERENCES
  • EXAM AND DISCUSSION QUESTIONS
  • NOTES

    This unit is contains far more information than can possibly be covered in a single lecture. The middle sections, D and E, contain a detailed technical review of a benchmark model. Depending on the abilities and interests of your students you may wish to omit these sections and move on to the description of the AGT benchmark in section G, or focus the lecture on the technical aspects and omit the descriptive example.

    UNIT 63 - BENCHMARKING

    A. INTRODUCTION

    Two types of benchmarking

    Benchmark script

    B. QUALITATIVE BENCHMARKS

    C. QUANTITATIVE BENCHMARKS

    Performance evaluation (PE)

    Subtasks for GIS PE

    Requirements for a quantitative benchmark

    GIS PE is more difficult

    D. EXAMPLE MODEL OF RESOURCE UTILIZATION

    Subtasks

    Products and data input

    Frequency required

    Execution of tasks

    Prediction

    Forecast

    Summary of phases of analysis

    E. APPLICATION OF MODEL

    Three phases of benchmark

    Qualitative benchmark

    Quantitative benchmark

    Model

    F. LIMITATIONS

    G. AGT BENCHMARK EXAMPLE

    Project Background

    REFERENCES

    Goodchild, M.F., 1987. "Application of a GIS benchmarking and workload estimation model," Papers and Proceedings of Applied Geography Conferences 10:1-6.

    Goodchild, M.F. and B.R. Rizzo, 1987. "Performance evaluation and workload estimation for geographic information systems," International Journal of Geographical Information Systems 1:67-76. Also appears in D.F. Marble, Editor, Proceedings of the Second International Symposium on Spatial Data Handling, Seattle, 497-509 (1986).

    Marble, D.F. and L. Sen, 1986. "The development of standardized benchmarks for spatial database systems," in D.F. Marble, Editor, Proceedings of the Second International Symposium on Spatial Data Handling, Seattle, 488-496.

    EXAM AND DISCUSSION QUESTIONS

    1. Discuss the Marble and Sen paper listed in the references, and the differences between its approach and that presented in this unit.

    2. How would you try to predict CPU utilization in the polygon overlay operation? What predictors would be suitable? How well would you expect them to perform based on your knowledge of algorithms for polygon overlay?

    3. Since a computer is a mechanical device, it should be perfectly predictable. Why, then, is it so difficult to forecast the resources used by a GIS task?

    4. Compare the approach to GIS applications benchmarking described in this unit with a standard description of computer performance evaluation, for example D. Ferrari, 1978, Computer Systems Performance Evaluation. Prentice Hall, Englewood Cliffs, NJ.

    5. In some parts of the computing industry, the need for benchmarks has been avoided through the development of standardized tests. For example such tests are used to compare the speed and throughput rates of numerically intensive supercomputers, and of general-purpose mainframes. Are such tests possible or appropriate in the GIS industry?

    6. GIS product definition exercise - 2 following pages.


    Back to Geography 370 Home Page

    Back to Geography 470 Home Page

    Back to GIS & Cartography Course Information Home Page



    Please send comments regarding content to: Brian Klinkenberg
    Please send comments regarding web-site problems to: The Techmaster
    Last Updated: August 30, 1997.