Step By Step Instructions

Step 1 | Step 2 | Step 3

Start Study Assessment - Step 1 – Instructions:
  1. Intervention (*required): Indicate the intervention/decision in question for which you are assessing the study or paper. For example, a specific drug, class of drug, a medical device, or a health care service.
  2. Paper Title (*required): Indicate the title of the paper in this field. [Note: You may indicate the complete title or an abbreviated one. This paper title is to help you to refer back to the paper in the future.]
  3. Paper citation/reference information (*required): In this field, it is suggested to indicate at least the first author on the paper and the journal the paper is published in and/or the PMID (PubMed ID). Example: Jansen, JP; Value Health 2011; 14:417-298, PMID: 23337210 [Note: This reference example is for your information only.]
  4. Upload Paper in PDF format (optional): This field is optional.  Papers uploaded here will be available through your study assessment history.
  5. Select Study Type (*required): Please select type of study to be assessed: indirect treatment comparison, modeling, prospective and retrospective observational studies.
    • Modeling study
       A modeling study uses equations to calculate the effect of interventions on outcomes of interest in populations and settings of interest. The equations are derived from clinical trials and observational studies.
    • Network meta-analysis/indirect treatment comparison study
       Indirect treatment comparison study
      A comparison of interventions that were studied in different randomized controlled trials. For example, the estimated treatment effect (e.g., odds ratio of response) of treatment B versus A from AB trials can be indirectly compared with the treatment effect of C versus A from AC trials to get an indirect estimate of the treatment effect of C versus B.
      Network meta-analysis study
      When the available randomized controlled trials concerning all the interventions of interest do not all compare the same interventions –each trial compares only a subset of the interventions of interest – it is possible to represent the evidence base as a network where all trials have at least one intervention in common with another. Such a network allows for indirect comparisons of interventions not studied in a head-to-head fashion. For example, the treatment effects from trials comparing treatments B relative to A (AB trials) and trials comparing treatments C relative to A (AC trials) can be pooled to obtain an indirect estimate for the comparison between treatments B and C.

    • Prospective observational study
       An observational study that is often longitudinal in nature, for which the consequential outcomes of interest occur after the study’s commencement (including creation of a study protocol and analysis plan, and study initiation). Patient exposure to any of the interventions being studied may have been recorded before the study initiation, such as when a prospective observational study uses an existing registry cohort. Exposure may include a pharmaceutical intervention, surgery, medical device, prescription, and decision made to treat.
    • Retrospective observational database study
       A study that looks backward in time to assess exposure and/or outcomes of interest that already occurred before the study was initiated. Often, retrospective studies utilize existing secondary data that have recorded exposure and outcome measures that have already occurred.
  6. Study Assessment History: By clicking “View my History,” a complete history of all the studies you assessed, including the paper title, reference information, a complete PDF of the study (if uploaded), the assessment report, and your rating is available. It provides access to your previous assessment reports to edit, print or save on your hard drive as a PDF. [Note: In your previous assessments, if a section is NOT complete it will show as “Not Answered.” To complete the section, click on edit and complete any missing questions.]
  7. Proceed to: Step 2 (Study Relevance)
Study (Paper) Relevance - Step 2 – Instructions:
  1. These interactive questionnaires will guide you to decide whether or not to include the study in your body of evidence. Based on your judgment, a study is either “sufficiently” or “insufficiently” relevant to the decision/intervention in question.
  2. To help with the understanding of the questions and the terms used, the following information is provided: 1) Definitions of some terms are accessed by hovering over the Underlined Terms; 2) Detailed explanations of some questions are accessed by using the link “Explanation” next to the question.
  3. To determine the relevancy of the study/paper to the decision/intervention in question, please follow the steps below:
    1. Begin by answering the questions (*required) as either: Yes, No, or Can't Answer. Reasons for “Can't Answer” include: –Not applicable, Not reported, Not enough information, Not enough training. Use the “note” link to add the reason for your answer (optional).
    2. Based on your answers, determine if the study has “Sufficient” or “Insufficient” relevance to the intervention/decision in question, and give your reasons using the “Add Notes” link.
    3. Tips:
      1. Use the “Add Notes” link whenever possible to explain the reason for your answers.
      2. You have the option to save an incomplete questionnaire and return at a later time to complete.
  4. Proceed to Step 3 (Study Credibility) or save and continue later. You can access your assessment(s) anytime from your “History” section.
  5. Note: If you determine the study is not relevant, there is no need to continue with the assessment
Study (Paper) Credibility - Step 3 – Instructions:
  1. The questionnaire uses a hierarchical checklist format (Domains - “Level One” questions – then sometimes, “Helper Questions” questions).
  2. These interactive questionnaires will guide you to decide whether or not to include the study in your body of evidence. Based on your judgment, a study is either “sufficiently” or “insufficiently” credible to include in your body of evidence.
  3. To help with the understanding of the questions and the terms used, the following information is provided: 1) Definitions of some terms are accessed by hovering over the Underlined Terms; 2) Detailed explanations of some questions are accessed by using the link “Explanation” next to the question.
  4. To determine the credibility of the study/paper to the decision/intervention in question, please follow the steps below:
    1. For each “Domain,” begin by answering the “Level One” questions (*required) as either: Yes, No, or Can't Answer. Reasons for “Can't Answer” include: –Not applicable, Not reported, Not enough information, Not enough training. Use the “note” link to add the reason for your answer (optional).
    2. Each “Level One” question may include an additional level of helper questions to help you accurately answer the question. 'Helper questions' are optional. You are not required to answer them, but it is recommended.
    3. Based on your answers to the questions within each specific “Domain,” provide an overall judgment of this domain effect on the study strength (*required). Is it considered a point of “Strength,” “Neutral,” or “Weakness”?
    4. Notes:
      1. Your answer to some of the questions could trigger a “ Fatal Flaw” indication. The “ Fatal Flaw” indication is NOT one of the options, but is generated by the system. It alerts you of the existence of a major weakness in the study related to that domain. You should seriously consider NOT adding this study to the body of evidence. However, you can overwrite the fatal flaw choice and check a different box if you choose to (not recommended).
      2. The “Fatal Flaw” choice needs to be overwritten at the “Level One” questions, before overwriting it at the “Domain” level.
      3. You are required to answer all “Level One” questions within a domain as well as the judgment question at the end of each “Domain” (see # 3 above) before making your decision on the credibility of the study.
    5. Based on your answers, determine if the study has “Sufficient” or “Insufficient” credibility to include in your body of evidence and give your reasons using “Add Notes” link.
    6. Tips:
      1. Use the “Add Notes” link whenever possible to explain the reason for your answers.
      2. Answer as many “Level Two” questions as possible in order to achieve the full potential of the assessment.
      3. You have the option to save an incomplete questionnaire and return at a later time to complete.
  5. Once you answered the overall credibility question, click “Finish.” Your assessment will be accessible from the “History” tab or “View my History” to print, view, edit and delete if you choose to.