Question) Fenton &Fleeter define measurement as follows: “Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe them according to clearly defined rules.” Kaner and Bond define measurement as follows: “Measurement is the empirical, objective assignment of numbers to attributes of objects or events (according to a rule derived from a model or theory) with the intent of describing them.” Working from either definition (your choice), consider this case: A professor decides to base the grades in her class on the height of her students. The taller the student, the higher the grade. Grades will be assigned on a 1-100 scale. The tallest student earns 100, all other students are given a score that matches the ratio (expressed as a percentage) of their height to the tallest student’s height.
i) Is this a measurement under this definition?
ii) Justify your answer to (a)
iii) Is this a surrogate measure? Explain why or why not.
iv) Briefly describe three problems with this proposed measure.
Question) Fenton & Fleeter define measurement as follows: “Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe them according to clearly defined rules.” Kaner and Bond define measurement as follows: “Measurement is the empirical, objective assignment of numbers to attributes of objects or events (according to a rule derived from a model or theory) with the intent of describing them.” Working from either definition (your choice), consider this case: We will measure the productivity of programmers by counting their lines (statements) of code.
i) Is this a measure under this definition? Explain why or why not
ii) Is this a surrogate measure? Explain why or why not.
iii) Name two strengths of this measure and explain why these are strengths
iv) Name two weaknesses of this measure and explain why these are weaknesses
Question) How do YOU recommend that we measure the productivity of programmers?
i) Describe the measure (or combination of measures) that you propose
ii) Is this a surrogate measure? Explain why or why not.
iii) Name two weaknesses of this measure and explain why these are weaknesses
iv) Why do you recommend this measure?
What makes it so good compared to What you consider to be the best alternatives. List and justify two or three strengths.
Question) In class, I presented an approach for describing and evaluating a metric that involved several factors (see the list below). Use this approach to evaluate the Defect Removal Efficiency (DRE) metric, which I will define as follows:
DRE = NF/NB and is estimated as NF / (NF+NR)
Where:
i) NB is the total number of bugs in a product
ii) NF is the number of bugs found before release, and
iii) NR is the number of different bugs reported by customers in the first year after the product is released
Here is the subset of the list of factors that I want you to consider for this exam:
i)Attribute you are trying to measure
ii) What are the instruments and how do you take the readings
iii) Likely sources of error
iv) Natural and foreseeable side effects
v) What rule or model allows us to map this measure to the attribute?
Question) In class, I presented an approach for describing and evaluating a metric that involved several factors (see the list below). Use this approach to evaluate the Development Productivity metric, which I will define as follows:
DP = NS/HOURS and is estimated as (NSN+NSM) / DH
Where:
i) DP is development productivity
ii) NS is number of statements
iii) HOURS is time worked
iv) NSN is the total number of new statements written this month
v) NSM is the number of statements written or replaced in maintenance this month, and
vi) DH is the number of hours worked this month by the development group members
Here is the subset of the list of factors that I want you to consider for this exam:
i) Attribute you are trying to measure
ii) What are the instruments and how do you take the readings
iii) Likely sources of error
iv) Natural and foreseeable side effects
v) What rule or model allows us to map this measure to the attribute?
If you don’t like this measure, what would you suggest that would fare better on these attributes?
Question) Let’s measure the management skills of project managers by comparing their schedule predictions to the actual results (how early or late, how much over or under budget).
Evaluate this measure in terms of Austin’s 2-party model and his 3-party model.
Question) Consider a user interface designer who is designing the front end of a database system.
(This is a business application, not a game or a work of art.) What is your operational definition of creativity for this person’s work? What surplus meaning does this definition miss? If we actually managed this person using this definition to recognize and reward their creativity, what distortions or dysfunction could we create as a result of ignoring the things involved in the surplus meaning?
Question) Imagine measuring the cost of a software development project (in time, money, or however you choose).
i) What is your measure (or combination of measures)?
ii) What are some potential biases in such a measurement? Describe three.
iii) Why do you think each of these three is a reasonable bias to be concerned about?
Question) Imagine measuring customer dissatisfaction with a business application in terms of the amount of time the customers waste when they use the product. How would you measure this? In particular:
i) What is your operational definition?
ii) What are two benefits of your approach? Think specifically in terms of things that contribute to the validity of the measurement. Explain your reasoning
iii) What are two risks of your approach? Think specifically in terms of realistic risks that threaten the validity of the measurement or that threaten distortion or dysfunction. Explain your reasoning.
Question) What is the cone of uncertainty? Describe Bossavit’s criticism of this concept? What’s your evaluation of his criticism?
Question) Consider Bossavit’s criticism of the research on programmer productivity. If you were going to do research on programmer productivity that would survive his criticism, what would you do? What measurements would you take? What are the validity issues with these? What would you do to make your reports credible?
Question) The representation condition assets that a measurement mapping, M, must map entities into numbers and empirical relations into numerical relations in such a way that the empirical relations preserve and are preserved by the numerical relations.
Give an example of a binary relation in the empirical world and three examples of statements that should be true, on the basis of this, in the numeric representation of this relation.
Question) In her paper, “Developing an Effective Metrics Program,” Rosenberg described a group of “sample goals, questions and metrics. The goals are general and could be adapted with minor modifications to any project development. Questions are derived to quantify the goals. The metrics needed to provide the answers to the questions are then chosen and shown in italics.”
Here is a goal from her paper, with associated questions and metrics:
i) GOAL: To predict the schedule in order to manage it
ii) QUESTION: What is the actual vs. expected effort level?
iii) METRIC: Effort (such as hours worked)
iv) QUESTION: What is the volatility of the requirements?
v) METRIC: Count of requirements, count of modifications to requirements
vi) QUESTION: What is the rate of module completion?
vii) METRIC: Count of modules completed
Use your knowledge of measurement dysfunction to critique this set of questions and metrics. In particular:
i) If you collected these metrics, would they provide you with answers to the questions?
Why or why not? What other information, if any, would you need?
ii) If you could answer these questions, could you accurately predict the schedule? Why or why not? What other information, if any, would you need?
iii) If you relied on these metrics, what aspects of your project do you think would be systematically under-managed? Explain your thinking.
Question) In her paper, “Developing an Effective Metrics Program,” Rosenberg described a group of “sample goals, questions and metrics. The goals are general and could be adapted with minor modifications to any project development. Questions are derived to quantify the goals. The metrics needed to provide the answers to the questions are then chosen and shown in italics.”
Here is a goal from her paper, with associated questions and metrics:
i) GOAL: The system must release on time with at least 90% of the errors located and removed
ii) QUESTION: When will 90% of the errors be found?
iii) METRIC: Effort (such as hours worked)
iv) METRIC: Errors (count errors detected)
v) QUESTION: What is the discrepancy rate of closure?
vi) METRIC: Errors (count errors detected)
vii) METRIC: Closure status of the errors
Use your knowledge of measurement dysfunction to critique this set of questions and metrics. In particular
i) If you collected these metrics, would they provide you with answers to the questions?
Why or why not? What other information, if any, would you need?
ii) If you could answer these questions, could you know whether at least 90% of the errors had been located and removed? Why or why not? What other information, if any, would you need?
iii) If you relied on these metrics, would any aspect of the project be systematically undermanaged or mismanaged? Explain your thinking.
Question) In her paper, “Developing an Effective Metrics Program,” Rosenberg described a group of “sample goals, questions and metrics. The goals are general and could be adapted with minor modifications to any project development. Questions are derived to quantify the goals. The metrics needed to provide the answers to the questions are then chosen and shown in italics.”
Here is a goal from her paper, with associated questions and metrics:
i) GOAL: Examine the product quality from the point of view of the customer
ii) QUESTION: What percentage of the modules exceeds the structure / architecture guidelines?
iii) METRIC: Size (such as LOC)
iv) METRIC: Complexity (such as Oviedo’s metric)
v) QUESTION: What modules is high risk?
vi) METRIC: Complexity (such as Oviedo’s metric) for each module
vii) METRIC: Size (such as LOC) for each module
viii) METRIC: Errors
Use your knowledge of measurement dysfunction to critique this set of questions and metrics. In particular
i) If you collected these metrics, would they provide you with answers to the questions?
Why or why not? What other information, if any, would you need?
ii) If you could answer these questions, could you accurately describe the quality from the point of view of the customer? Why or why not? What other information, if any, would you need?
iii) If you relied on these metrics, would any aspect of the project be systematically undermanaged or mismanaged? Explain your thinking.
Question) Suppose that you wanted to PREDICT THE DIFFICULTY of a PROGRAMMING PROJECT in order to HIRE APPROPRIATELY for it. Suppose too that your company has done lots of projects and kept lots of raw data that you can mine.
i) Use the Goal / Question / Metric approach to choose some metrics that you could appropriately use to develop your prediction(s). Briefly describe between 4 and 7 such metrics.
ii) Pick two of your suggested metrics and for each one, state two good reasons for thinking it would be valid and useful. State one reason for thinking that it would be invalid or no useful.