DesignWIKI

Fil Salustri's Design Site

Site Tools


research:conducting_research

Conducting Research

Research is a particular activity with a methodology. Here are some notes about doing research.

Introduction

Research is a disciplined, directed way to create knowledge. Knowledge leads to understanding, which puts us in a better position to change how things are to be better. Put another way, research is an activity you are motivated to do to solve a problem.

  • But at the beginning of a research activity (e.g. doing a graduate thesis) the problem is usually defined only vaguely.

There are many different ways to do research. The information here focuses on research pertinent to the phenomenological1) study of design.

Getting from a vague sense of a problem to a relatively well-bounded research question is the first step.

Research question

A research question is a statement, in the form of a question, that identifies the phenomenon to be studied.

Research questions apply to specific topics (e.g. diagrams in design, FEA for analysis of bone, underwater robots, etc.). We assume here you have already established a suitable topic with your academic adviser.

Three tasks in generating a good research question:

  1. Get some experience.
    • That is, read a lot of the available literature.
    • Try to get involved in activities pertinent to your general topic.
      • If you're studying collaboration, get involved in collaborative projects.
      • If you're studying design, go design something.
    • No matter how you get the experience, keep careful notes of everything that you do.
      • If you're reading things, make sure you keep notes of every paper you read, whether it's good or not.
      • If you're doing other activities, keep a journal of those activities, including your reflections on the activity itself.
  2. Create your own mental model of that information.
    • A recommended way to do this is with concept maps.
    • In such a concept map:
      • nodes would capture all the concepts you found in all your experience/reading; and
      • links would capture relationships between the concepts.
      • You can include your own concepts and relationships, beyond those you learnt.
      • You can identify your own concepts and relationships just by using different colours for those nodes and links.
  3. Build a research question in the context of your mental model (your concept map). To help you do this, consider these questions.
    • What are the concepts and relationships that you find most interesting?
    • What open issues have not been addressed by the literature? That is, are there any interesting holes (missing information) in your concept map?
    • Do you see a different way of thinking about some concepts and relationships, that haven't been covered by the literature?
    • What questions do you have arising from the concept map? What do you wish you'd have read about or learnt?

Also, as you carry out these tasks, you should ask yourself (and answer!) these questions:

  • Do I know the field and its literature well?
    • If not, how do I get to know it well?
  • What are the important research questions in my field?
    • Each time you read a paper, ask yourself What research question did the author(s) answer with this paper?
      • Make a list of those research questions.
      • The more you read, the more you should find papers working on the same or similar research questions.
    • Research questions in the literature will change over time.
      • Track the year of publication when you track the research questions in other people's work. This will give you a good sense of how the field your working in is progressing.
    • You can create a concept map of the research questions.
      • You can use, say, the horizontal axis to represent time, and the links between nodes to represent your sense of the relationships between different questions.
  • What areas need further exploration?
    • i.e. where are there “holes” in your concept map of research questions.
  • Could my study fill a gap? Lead to greater understanding?
    • How big is the gap? Can you imagine filling it within the limits of your research time?
  • Has a great deal of research already been conducted in this topic area?
  • Has this study been done before? If so, is there room for improvement?
  • Is the timing right for this question to be answered? Is it a hot topic, or is it becoming obsolete?
  • Would funding sources be interested?
    • This is something best discussed carefully with your academic adviser.
  • If I am proposing a service program, is the target community interested?
  • Most importantly, will my study have a significant impact on the field?

Characteristics of a good research question:

  • fairly specific
    • this means there should be only 2 or 3 general variables in your research question
  • definitely unanswered - a new question
    • For a PhD, this must certainly be true at the end of the literature survey portion of your studies.
    • For a homework assignment at the undergraduate level, this must be true with expect only to the amount and breadth of reviewed literature, within reason for the assignment.
  • always defined with respect to a known context, including theoretical underpinnings
    • Example: a research question like Can FMEA be used reliably as a design-analytic tool at the systems level? assumes there is readily available information about FMEA, about design analysis, about systems design and analysis, and about measuring methodological reliability. The reader of a document dealing with this research question would expect all this contextual information to be more or less explained in the document.
  • is feasible within allotted time, cost, and ethical bounds.
  • can use relative terms (e.g. “better”, “worse”, “improved”).
  • must use crisply defined (or definable) concepts.

While you're developing your research question, remember to:

  • Take breaks to get away from the matter and give your mind a rest.
    • You'll come back to the matter of finding a research question with a fresh perspective if you take breaks.
  • Talk it out with your academic adviser, other colleagues, and friends.
    • Having to express your thoughts to others will almost always help you clarify your own thinking.
  • Be aware of causality implied by a possible research question, which may only be an artifact of your question and *not* a property of the research question.

There are different kinds of research questions, not all of which will make sense for particular topics. Still, it can be useful to get a couple of research questions of each kind, and running them by your academic adviser.

Factual research questions: these are popular in “scientific” studies, where you simply look for causal relationships and describe them. For example:

  • Do concept maps help design engineers collaborate during early designing tasks?
  • Is FMEA a beneficial design-analytic tool at the systems level?
  • What have been the trends in adoption of automotive technologies over the past 20 years?

Paradoxical research questions: these describe an apparently contradictory situation and suggest you will resolve the contradiction. For example:

  • Why do North American consumers continue to demand increased product quality but continue to buy the “cheapest” products?
  • If design engineers use diagrammatic visualizations to help them understand engineering problems, why are requirements always written as text-only documents?
  • How can engineers make better products using computer tools, if engineers know increasingly less about how the software they use works?

Hypothetical research questions: these help us understand why things are the way they are now, by considering how things would be different if something in the past had happened differently. These kinds of questions are not often used in design research. For example:

  • How would the world have been if Ford had not popularized mass production of automobiles?
  • Would Warner Brothers still exist today if Bugs Bunny had been a female cartoon figure?
  • How would modern life differ if perspective drawing had been invented before the Middle Ages instead of after?

Predictive research questions: these help us make decisions by constructing “scenarios” of how things might be in the future. For example:

  • What will North American society be like in 50 years if we continue to use petroleum-based products at current day rates?
  • How might Apple's significant reduction in price of it's iPhone affect Blackberry sales?
  • What will our computer-based society do if a giant solar storm knocks out electronics globally within 5 years?

Problem-solving research questions: these propose specific solutions to existent problems. For example:

  • Will programming “instinct” overcome bottlenecks in robotic AI performance?
  • Can a 40-noded spring element address the FEA needs of modellers of soft-tissue biomechanics?
  • Can mereotopology provide the formal foundations for a new theory of artifacts?

Comparative research questions: these identify (usually) two alternatives to a situation and compare the alternatives in actual practise. For example:

  • Are paper bags more “sustainable” than plastic bags for groceries in varied North American markets?
  • Is a program of supported employment more effective (than no program at all) at keeping newly employed persons on the job?
  • Does “bioethanol” produce less greenhouse gas than gasoline, when one accounts for the production of both?

Judgemental research questions: these require rational and rhetorical arguments to answer the question instead of scientific or predictive methods (that is, no definitive answer in a scientific sense is possible). For example:

  • Is globalization is harming Ontario SMEs?
  • What is the impact of carbon-fibre based materials on job satisfaction of assembly workers in the North American aerospace sector?
  • To what extent is engineering design related to design in the arts?

Bad research questions include the following. (Can you tell why they're bad research questions?)

  • What can be done to prevent substance abuse?
  • Can CAD improve the efficiency of design engineers?
  • How can design be improved?

Research questions should always be quite specific, but even then there is great flexibility in just how specific your research question should be.

  • Specificity will depend on how much time you have to answer the question, resources available to help you, and input from your academic adviser.

You can always make your research question more specific, or less specific. Here are two examples:

Topic Frank Lloyd Wright and modern architecture
A research question How has Frank Lloyd Wright influenced modern architecture?
A more specific research question What design principles used by Frank Lloyd Wright are common in contemporary homes?
Topic nutrition, children, health education, health risks
A research question How does nutrition education help children?
A more specific research question What are the major health risks related to diet for school aged children?
Another specific research question What are the benefits of nutrition education for school children?

Causal analysis

Many research questions imply causality - cause and effect - between the concepts in it.

Remember, though, that correlation alone does not imply causation.

For example, consider the research question: Do concept maps help design engineers collaborate during early designing tasks?

  • There is implied causality between the use of concept maps (the cause) and improved collaboration (the effect).

But since this is only a research question, you don't know for sure if the causality is real, or just an artifact of how you phrased the research question.

  • So, verifying causality becomes part of the hypotheses and specific aims (these are described below) of your research.

There are three main criteria for inferring a cause and effect relationship:

  1. variation in the cause must correlate to variation in the effect;
  2. causes must occur before effects; and
  3. there must not be any other (reasonable) explanation for the proposed causality.

So when you are thinking about causal relationships in your research questions, you have to consider how you will validate that causality, as defined above. This too becomes part of your research project.

Causal analysis can be very hard and take a long time. Causal analysis and validation could be your entire research project! You might have to conduct many experiments to discount alternative causal relations, or conduct a variety of analyses - mathematical or otherwise.

In the example above – Do concept maps help design engineers collaborate during early designing tasks? – you would have to search for other phenomena associated with the introduction of concept maps to a group of engineers that might cause them to collaborate, regardless of the use of concept maps themselves.

  • One fairly common way of getting around this is to repeat your “experiments” with significantly different settings and groups of participants.
    • If you vary everything else, and yet still find a good correlation between concept maps and collaboration, then you can argue that you've eliminated the alternative causations.
    • You can see how much work would be involved in doing this.
  • There are many other ways to (try to) validate causality.

Causality can feed back on itself. If concept maps improve collaboration, the increased collaboration may drive further use of concept maps, which further increases collaboration - to a point. You have to be wary of feedback, both positive and negative, because it makes the whole system unstable. For instances, designers can collaborate too much and end up not getting any work done, in which case collaboration becomes a cost and not a benefit.

  • This tends to mean that “experiments” must be repeated varying, in this case, rates of usage of concept maps, to try to identify if and how feedback occurs, so that it can be accounted for in the causal analysis.

Going from research question to hypothesis

The second step of a research project is turning the research question into a hypothesis (or a small set of related hypotheses). Rarely do specific projects generate more than three hypotheses for a given research question.

A hypothesis is a specific prediction about the nature and direction of the relationship between two or more variables.

A good research question should lead almost directly to at least one hypothesis. For example:

  • Research question: What resources would be helpful to new and minority drug abuse researchers?
  • Hypothesis: A grant writing tutorial would be helpful to new and minority drug abuse researchers. Those researchers who utilize an on-line grant writing tutorial will have higher priority scores on their next grant application than those who do not.

A good hypothesis has certain characteristics, including:

  • Give insight into the research question;
  • Are testable and measurable by the proposed experiments; and
  • Clearly identifying a proposed relationship between very few (usually only 2) variables for which data can reasonably be expected to be gathered or found.

Other good hypotheses include:

  • Those researchers who utilize an online grant writing tutorial will have higher priority scores on their next grant application than those who do not.

You must be able to justify your hypotheses. That means:

  • You have to be able to explain how you developed each hypothesis - where it came from;
  • You have to demonstrate that you at least searched for alternative hypotheses.
    • For cases where alternatives were found, you have to be able to explain your selection process.

Going from hypothesis to specific aim

A specific aim is a description of the steps you will take to prove or disprove your hypothesis.

A specific aim must be:

  • testable, with measurable results;
  • every specific aim must map to one hypothesis;
  • every aim is feasible within the limits of your research project.

An example of a good specific aim is given below, with respect to a hypothesis.

  • Hypothesis: A grant writing tutorial would be helpful to new and minority drug abuse researchers. Those researchers who utilize an on-line grant writing tutorial will have higher priority scores on their next grant application than those who do not.
  • Specific aim: Conduct a rigorous empirical evaluation of the on-line grant writing tutorial, comparing outcome and process measures from two groups-those with exposure to the tutorial, and those without.

Writing up research

Dissertations

Prefer a breadth-first presentation: Your research can be thought of as having two dimensions: breadth of coverage, including the different aspects and areas of it, and depth of coverage, including the various levels of detail to which each aspect and area is covered.

It is usually easier for the general reader to understand a breadth-first presentation, where each section or chapter explains matters at a lower/finer level of details than the last.

The breadth should extend well outside the reported work in the Introduction (but obviously not to any significant detail), to show how your research fits in with the rest of the reported work (which you will cover in the Literature Review).

Merging Examiner Comments

After an oral examination, all the issues, questions, and comments of the examiners have to be merged and integrated into the dissertation. This can be very challenging, especially for very long, complex dissertations. Different examiners will require different changes. Sometimes, the requested changes will directly contradict one another; other times, one change will necessitate various others. In either case, tracking exactly what has to be done to the dissertation can be an onerous project management task entirely unrelated to the research that you did.

One of my PhD students, David Fourney, came up with a chart-based way to manage all those changes that I found greatly simplified the project management aspects and let both David and I focus on making sure that the dissertation was as robust as possible. I describe this method here, in the hope it will help others.

This method works best in Google Spreadsheets or Excel.

Create a table with columns as shown below (and an entirely imaginary sample row is given too).

STATUS PAGE REVIEWER COMMENT ACTION SUPERVISOR COMMENT
Active 53 Some of the p-values reported indicate non-significant results. This in itself isn't a problem, but you have to admit that the results were not significant and explain what that means for the work overall. Rewrite discussion for the chapter and overall thesis discussion. Identify which results were significant and which weren't. Add to future work to study the non-significant results. Hang on. Fisher basically invented 95% confidence level thing; it's completely arbitrary. We need to talk this out. I think it's better to calculate the confidence interval rather than set it, and then use the analysis to rank your results from most to least robust, statistically.

First, you should go through all the examiners' comments and create an initial table. You will start with only the first three columns filled in. Try to capture the examiners' comments verbatim to preserve as much of the original intent as possible.

The status of each entry can be one of three: Open (not yet started), Active (an item you're currently working on), or Done (having completed thesis updates to address the concern).

Then, go through the table carefully, merging rows that all deal with a given concern. Distinguish between different examiners' comments when you do the merge.

Share the spreadsheet with your supervisor(s) and give them a chance to review it. The table can be used as a source document to guide discussion with your supervisor. But perhaps more importantly, it lets you and your supervisor communicate asynchronously without having to always meet face to face.

Be scrupulous about keeping the table up to date. Any time any progress is made, make sure update the Status and Action cells to reflect the changes. You can also use the Action cells to create a “todo list” of sorts for particularly complex changes.

Carefully track your progress. Note that it's very easy to sort the spreadsheet by the status column, which will group all the open, active, and completed rows together. You can even easily create counters that will total up how many open, active, and completed items you have with the COUNTIF spreadsheet function - which will give you a sense of your progress.

That's basically all there is to it.

10 Principles for Reference and Citation

These principles are attributed to Prof. K. Friedman.

  1. Use citations constructively to substantiate the argument of an article.
  2. Use citations creatively to advance the argument of an article.
  3. The author must argue a case in the explicit narrative of the article. External sources support an argument. External support for an argument cannot replace the author’s own argument. Do not confuse the two.
  4. Use precise, fine-grained references that permit the reader to locate quoted material at the exact location in the source document. Fine-grained references allow the reader to question and challenge cited sources.
  5. Treat direct quotations, indirect quotations, and paraphrases the same way. Give precise references for all quotations and cited sources. This helps the reader while building and supporting the knowledge of the field.
  6. Always review and re-read cited passages from referenced sources. This ensures correct quotes and accurate paraphrasing while helping the author to develop the meaning of the source text effectively. It also allows the author to reflect on text surrounding quoted material for added depth and possible use.
  7. Never use second-hand references from other authors. Always check cited sources first hand.
  8. Never use loose or vague references.
  9. Each item cited in the text must appear in the reference list. Every item in the reference list must appear in the text.
  10. Each source cited in the text requires an appropriate in-text citation and an entry in the reference list. Every entry in the reference list must be complete. All citations and all references must use the same style. All citations and references must be complete and consistent to be correct.

See Also

1)
Phenomenon: the object of a person's perceptions.
research/conducting_research.txt · Last modified: 2020.03.12 13:30 (external edit)