Conducting Needs Assessments: A Multidisciplinary Approach

Books

Fernando I. Soriano

  • Citations
  • Add to My List
  • Text Size

  • Chapters
  • Front Matter
  • Back Matter
  • Subject Index
  • Part I: Foundational Knowledge

    Part II: Planning

    Part III: Methods

    Part IV: Approaching Participants

    Part V: Finalizing the Study

  • Copyright

    View Copyright Page

    Preface

    My first professional job after I obtained my PhD in psychology was as a research psychologist for the US Navy. It was during my tenure with the Navy Personnel Research and Development Center in San Diego that I first became interested and involved in conducting needs assessments. With the establishment of Navy Family Service Centers at Navy bases worldwide, concomitant with looming military budget cuts, base commanders were being asked to undertake needs assessments of military personnel and their dependents. At that time, the military was dramatically expanding its interest in military families, because studies were suggesting a link between their well-being and such military concerns as job performance, readiness, and retention. Similarly, my subsequent work with community-based agencies involved needs assessments. Altogether, these experiences demonstrated to me the importance of needs assessments and pointed to the lack of information available about this powerful assessment concept and tool. The success of the first edition of this book confirmed my suspicion that there is indeed a need for a book that articulates common methods of assessing needs.

    The decision to provide the specific information in this book came from my interaction with those needing to conduct and benefiting from needs assessments. Like the first edition, this one is written to be easy to read. Yet it is also technically sophisticated and comprehensive enough to address such critical questions as knowing the first steps to take in planning an assessment, determining sample sizes, conducting appropriate analyses, and using the best formats for reporting findings.

    It is my hope that readers will find this book as helpful, practical, and approachable as it was intended to be. There are several features of this edition worth noting. The first three chapters comprise the foundational knowledge necessary to undertake a realistic and effective needs assessment. Chapter 2 considers the sociopolitical realities found in many communities. Chapter 3 discusses the special considerations and measurement issues one must consider when working with culturally and socially diverse populations of both genders. From the start, the United States has been multicultural and socially diverse, and it is now more diverse than ever. Conducting successful needs assessments necessitates taking into consideration the inherent diversity of target populations—whether based on culture, gender, income, education, or regional differences.

    Many people helped with this book. First and foremost, I want to thank all who provided me with encouragement and support throughout its creation.

    I wish to thank Sissy, my companion, for being such a support and encouragement to me.

    I also particularly wish to acknowledge the support of my late mother, Ignacia Soriano, who was always ready to validate my efforts, no matter what they were—even if she did not understand or care to understand them. Whatever I did, she was always proud of me and always encouraged me. I miss her dearly, but feel her presence and positive influence in my life every day nonetheless.

    Also important are my boys, Anthony and Fernando, who served as an inspiration to me. They are so funny, fun, and entertaining and only sometimes challenging.

    I thank God for being blessed with these wonderful lifelong gifts of friends and family!

    I also wish to acknowledge the many military and civilian community leaders who helped me to better understand what type of information is lacking on needs assessments. Their influence continues to be felt by me.

    I wish to express my deep appreciation to series editor Armand Lauffer from Sage, who was abundantly patient and encouraging in the journey of writing the first edition and now this one. Last but certainly not least, I am thankful to Kassie Graves, this book's aquisitions editor, who believed in me and was patient as I worked on this edition. Without her encouragement and able assistance, this new edition would not exist—period. To her I owe great gratitude for skillful editorial assistance, and she helped accomplish the many detailed editorial and publication tasks involved. She also made the process very pleasant. This book, minus its shortcomings, is a tribute to Lauffer and Graves.

    FernandoI.Soriano

    Introduction

    “Anthony, things are so different now than in the past. Remember when we were assured of funding year after year? You were just out of graduate school, and I hired you as my associate director, and I didn't need you to have more skills than just the promise of learning quickly how to help me oversee the many programs and services for our youth and families. Now I honestly don't know what to do. The same grant proposals are being turned down, and we're about ready to lay off over 30% of our staff for lack of funding.”

    “But Nano, I've gone to the meetings too. You keep thinking that we're in a sinking ship that has no hope, but that's not how I see it. I keep telling you that the agencies that are getting funding these days are those that show evidence of the need for their services! They are the ones doing needs assessments.”

    Funding for human services has always vacillated depending on the economy and policy priorities in public and private sectors. However, the “great recession” has been unprecedented in the depth of its impact on all sectors of the economy. For example, it is estimated that at the height of the economic downturn in 2009, over $2 billion was taken away from California alone for human services at a time when need was growing to unprecedented levels (CWDA-CSAC, 2009). As never before, human service providers are confronted with the need to prove that their services are indeed needed. In the past, this proof could have been verification that services were being rendered. Now service providers are likely to be required to demonstrate a clear awareness of their target populations' need for programs and services. The above vignette illustrates the increasing demand for empirical data to justify funding for human service programs. As a result, many representatives of the social and human service fields will be involved in commissioning, conducting, or utilizing information from needs assessments—if not for justifying services, then for determining what services should be offered.

    The term needs assessment appears simple enough to understand. It refers to the process of assessing needs. To many, needs assessments refer to the collection of data bearing on the need for services, products, or information. In fact, it is the conceptually appealing nature of the term that makes its frequent application attractive. Unfortunately, the process of conducting needs assessments is far from intuitive. Rather, it is technical and sophisticated. And it requires an awareness of the scope and significance of the information collected.

    According to Petersen and Alexander (2001), “the role of needs assessment is to identify and also address needs” (p. 4). Kaufman and English (1979) called it “a tool for determining valid and useful problems which are philosophically as well as practically sound. It keeps us from running down more blind educational alleys, from using time, dollars and people in attempted solutions which do not work” (p. 31). From these definitions, we can conclude that needs assessments enable us to obtain valid and reliable information, which helps us to better target our services and efforts.

    Private businesses readily recognize the importance of evaluating the need for their products or services. A lack of need for what they offer translates into no sales. Their immediate criteria for success are sales and profits. On the other hand, in the human services realm, and in particular among not-for-profit organizations, the volume of sales and profits is not the driving force. Typically, the criterion for success is providing the services they recognize are needed in their catchment area. A need for social programs or human services is too often only assumed or believed evident, and efforts to measure it are lacking. Among those who attempt to conduct needs assessments, many lack the requisite knowledge of how to do so. An all too common result is the gathering of uninterpretable or unrepresentative information.

    The demand for a better understanding of needs assessments is increasing, largely due to fewer monies being available for human service providers. A lack of available resources has led to increased competition for funding among services. Subjective appeals from service providers for funding will no longer be sufficient. To ensure that limited monies are appropriated to the most useful services, funders are demanding clearer justification for programming budgets. The empirical demonstration of a need for services will undoubtedly continue to be a requirement for years to come.

    Many funding agencies and foundations are requiring that applications for funding include results from needs assessments that demonstrate a clear need for the proposed programs or services. This requirement has led to a significant increase in the number of needs assessments being conducted for and by agencies. However, as those who are engaged in assessments are finding, the term needs assessment is often easier to comprehend than to apply. Moreover, serious measurement risks can result from conducting improper needs assessments. A needs assessment conducted in the general community, for example, may reveal a need for drug abuse treatment services, but it may not be specific enough to point out the particular service needs of certain subpopulations such as the homeless, the highly transient, ethnic minority groups, non-English speakers, and so on. Needs assessments can be misleading when not carefully planned and executed.

    There are myriad reasons for conducting needs assessments, whether to meet requirements imposed by entities external to an organization or to comply with internal requests. The following are among the most common reasons:

    • Provide justification for funding.
    • Comply with regulations or laws that mandate needs assessments.
    • Inform resource allocation and decision making—determine the best use of limited resources.
    • Assess the needs of specific, underserved subpopulations.
    • Provide information that is part of program evaluations.

    As mentioned earlier, funding agencies are increasingly demanding empirical justification for the financial support they provide. Furthermore, a growing number of state and federal agencies, as well as private foundations, mandate that funded agencies and programs conduct periodic needs assessments to ensure that funds are being used for services that are most needed.

    The mandate for program justification is not the only driving force behind the implementation of need assessments. Interest in such studies comes from various sources and situations. Successful, established service programs often recognize that needs assessments can help them to prioritize their efforts for the greatest possible programmatic effect. Results from needs assessments are used to make decisions about internal programming and resource allocation. Tragic social upheavals, such as those due to Hurricane Katrina in 2005, also cause agencies and programs to reevaluate their effectiveness in meeting the needs of underserved communities. Well-targeted needs assessments can do much to identify the needs of frequently overlooked subpopulations, including ethnic minorities, adolescents, women, single parents, the elderly, and the homeless.

    Given the right planning, needs assessments can also form part of a program evaluation effort. Used in this way, needs assessments are conducted both before and after programmatic efforts are in place. The directors of a drug abuse prevention program may, for example, decide that the effectiveness of their community awareness campaigns can be reflected in both the reduction of use and in the perceived need for drug abuse prevention services. Hence, they may decide that changes suggested by responses to pre- and post-needs assessments will form part of their program evaluation. They can also use the results from each assessment to prioritize their services.

    Agencies and programs commonly have multiple reasons for conducting or commissioning needs assessments—some externally imposed, others internally derived. In many cases, needs assessments that are implemented at the request of an external agency would not otherwise be initiated. The perceived imposition of having to conduct an assessment leads to the term engendering strongly negative connotations—it is viewed as a threat, burden, or requirement. Consequently, programs required to conduct needs assessments by outside agencies are often the least motivated to utilize the information gained from the reports for their own purposes. In contrast, agencies and programs that recognize the value of empirical information are the most likely to benefit from assessments.

    Whether initiated internally or externally, whether viewed as a hassle or a help, needs assessments can help ensure programmatic success and efficiency by providing timely information about the communities being served. Human service programs and organizations are, by virtue of their mission, dynamic and evolving. Their effectiveness and survival increasingly depend on using objective data to guide their efforts. This book is designed to help agencies, programs, researchers, and others better understand needs assessment methodologies.

    This book is suitable for use in courses and workshops and as part of the reference material of human service agencies and organizations. It is designed to provide a basic but working understanding of needs assessments. I will focus on the methods commonly used in conducting needs assessments within the human or social services fields. This book is structured as a practical, how-to resource, guiding the reader through the steps required to undertake a comprehensive needs assessment. It begins with the formulation of realistic objectives and ends with a discussion of effective reporting formats. Each chapter contains examples, suggestions, and exercises for developing basic research competencies. A needs assessment guide, in Appendix A, guides the user through each conceptual and empirical step outlined in the book.

    This book is meant to complement other books on needs assessments that take a more comprehensive approach (Altschuld & Witkin, 2000) or discuss assessments' application in diverse areas (Reviere, Berkowitz, Carter, & Ferguson, 1996) or in specific fields, such as in health (Bosworth, 2000) and health education (Gilmore & Campbell, 2005). It builds on early volumes in this series by Sage (Berger & Patchner, 1988a, 1988b; Blythe & Tripodi, 1989; Coley & Scheinberg, 1990; Kettner, Moroney, & Martin, 1990; Lauffer, 1982; Neuber, 1980; Schaefer, 1987). The goal, particularly of this new edition, has been to present a practical and contemporary discussion of complex topics that are not discussed in other books, such as the sociopolitical and cultural context of communities, that can pose challenges to the conduct of needs assessments.

    As with the earlier edition, a special feature of this book is its multidisciplinary approach to needs assessments. Other publications have typically been grounded in a particular discipline, such as business or mental health. Another advantage of this book is that it assumes a candid and pragmatic approach to conducting needs assessments. Too often, those commissioned to conduct needs assessments fail to understand the organizational limitations on utilizing new information; thus, their suggestions assume organizational change is both easy and welcome. New information may not be accepted by the agency, however, for reasons that vary from political motivations (maintenance of the status quo) to pragmatic reasons such as budgetary constraints.

    Overview

    The presentation of topics in the book's chapters follows a conceptual and practical sequence that plots out the steps for conducting needs assessments. The book is divided into five sections. Chapters in Section I, Foundational Knowledge, provide baseline knowledge on the understanding of the nature of needs assessments and of the context within which they are conducted that determine their success. Specifically in this first section, Chapter 1 presents conceptual and theoretical foundational knowledge. The importance of theory is emphasized, and needs assessments are clearly defined and distinguished from other assessment and evaluation tools. Chapter 2 on sociopolitical and economic considerations presents the realities of social and political sensitivities and controversies within communities and society that agencies have to consider, because they can determine the viability, success, and impact of needs assessments. The chapter offers ways of working around these while keeping the focus on addressing the needs of vulnerable and underserved populations. Compared to the earlier edition, Chapter 3 is an expanded consideration of social, gender, and cultural factors that also need to be considered. It alerts the reader to the importance of these variables and offers ways of measuring and including them in needs assessments.

    One chapter comprises Section II, Planning. In Chapter 4, the reader is taken through a step-by-step process that begins with developing a clear focus for the needs assessment and ends with selecting the general type of needs assessment methodology, whether qualitative, quantitative, or mixed.

    In Section III, Methods, Chapters 5 through 8 introduce the reader to commonly used quantitative and qualitative methods and suggest the most appropriate techniques for analyzing data generated using the various methodologies.

    In Section IV, Approaching Participants, Chapter 9 helps the reader understand socially, culturally, and ethically sensitive ways to recruit study participants. A treatment of privacy issues, also discussed in Chapter 9, is a new addition in this edition. Federal regulations, such as the Health Insurance Portability and Accountability Act (HIPAA), are discussed here.

    Also encompassing a single chapter (Chapter 10), Section V, Finalizing the Study, focuses on helping the reader present the needs assessment findings in a final report.

    Finally, the Needs Assessment Guide described in this book is included in Appendix A, and the Pan-Acculturation Scale (PAN) is provided in Appendix B.

    Dedication

    This edition is dedicated to the memory of my parents, Marcelo and particularly Ignacia Soriano, who not long ago passed away. I most certainly would not have published any paper or book without the unfailing, unconditional faith and high expectations they had for me. Boy, do I miss you both! Can't believe I'm an orphan now!

  • Appendix A: Needs Assessment Guide

    Instructions

    This guide is designed to help you think about the information needed to conduct a needs assessment. The guide does not focus on a particular subject or field. It can be used for designing needs assessments in most social service and human service fields. The guide is divided into the following eight sections:

    • Purpose and Objectives
    • Roles and Responsibilities
    • Target Population and Subgroups
    • Stakeholders
    • Resource Availability
    • Use of the Information
    • Decision-Making Guide
    • General Checklist

    Answer the questions in each section as completely as possible and follow additional instructions in the guide. Place a check mark next to the questions you cannot answer at this time. Seek out the information needed to answer any questions left blank before conducting a needs assessment.

    At the end of the guide is a general checklist that can help you identify the sections in which all questions have been answered. This guide is not all-encompassing and should not be used alone. You will need to refer to other sources of information, such as the chapters in this book and the reading material in the reference section, before making final decisions about the design and implementation of a needs assessment.

    I. Purpose and Objectives
    • General Purpose: What is the purpose of conducting a needs assessment?
    • Objectives: What are the key questions for which you need answers? The key questions determine the scope of a needs assessment and are directed at the purpose of the study.

    Caution: These questions will guide the data collected. Key Questions:

    II. Roles and Responsibilities

    The following questions refer to those who are responsible for aspects of the needs assessment.

    General Responsibility
    • Who is responsible for seeing that a needs assessment is conducted?

      NamePosition

      (More than one person, if appropriate.)

    Specific Responsibilities

    List individuals responsible for the following components of the needs assessment:

    • General planning and coordinating (all aspects of the study):

      NamePosition

      (More than one person, if appropriate.)

    • Designing the study:

      NamePosition

      (More than one person, if appropriate.)

    • Collecting the data:

      NamePosition

      (More than one person, if appropriate.)

    • Coding or preparing the data for analysis:

      NamePosition

      (More than one person, if appropriate.)

    • Analyzing the data (i.e., tabulating or noting the results without interpreting them):

      NamePosition

      (More than one person, if appropriate.)

    • Interpreting the results (i.e., giving meaning to the data):

      NamePosition

      (More than one person, if appropriate.)

    • Reporting the results:

      NamePosition

      (More than one person, if appropriate.)

    Stop and consider those people delegated responsibilities. Ask yourself these questions:

    Are they being granted release time to accomplish these responsibilities?

    Do they have the needed interest and skills, or are they able and willing to acquire them, to accomplish their assigned task(s)?

    If the needs assessment is a group effort, are members of this group willing and able to work together, do they have the opportunity to do so, and and can they communicate with one another well enough to accomplish the entire task?

    If any responses to these questions are “no,” reassign personnel and responsibilities or consider contracting the needs assessment or parts of it to outside experts (e.g., university researchers, private research contractors, parent agency, consultants, etc.).

    Oversight Committee

    Form an oversight committee made up of internal and external members of the organization. At least one member of the target community should be a part of this committee, and the committee should reflect the diversity of the target population.

    • Describe the membership of the oversight committee.

      Number of members:

      Number of members from the target population:

      Number of members who are ethnic minorities reflective of target population:

    • Indicate how often the committee will meet.

      Will the oversight committee be used to oversee:

    • Ethics?

      YesNoNot sure (Please determine.)

      If no, explain who will oversee ethics:

    • Wording of questionnaire?

      YesNoNot sure (Please determine.)

      If no, explain who will oversee wording of questionnaire:

    • Cultural sensitivity and relevance?

      YesNoNot sure (Please determine.)

      If no, explain who will oversee cultural sensitivity and relevance:

    • General appropriateness of study design and plans to implement it?

      YesNoNot sure (Please determine.)

      If no, explain who will oversee study design:

    III. Target Population and Subgroups

    These questions pertain to the targeted population for the needs assessment and the relevant subgroups.

    • What are the physical boundaries delineating the target area?
    • Describe the target population. Use demographic or other key characteristics important for defining who is and is not a member of the target population.
    • Is it important for you specifically to compare needs among ethnic groups?

      YesNo

    • If the answer to (c) is yes, check off the ethnic groups from which you need information:
      • Anglo Americans
      • Hispanics/Latinos
      • African Americans
      • Native Americans
      • Asian Americans
      • Others:
    • For each subgroup marked, indicate how you will determine ethnic group membership (e.g., self-identification).
    • Is it important for you specifically to compare needs between gender groups?

      YesNo

    • If the answer to (f) is yes, will you want 50% of each in your sample?

      YesNo

    • Is it important for you to compare subgroups differing in education (e.g., high school vs. college graduates)?

      YesNo

    • If the answer to (h) is yes, describe the different education subgroups and how you will define each.
    • Is it important for you to compare subgroups differing in income levels (e.g., family income under $10,000 vs. more than this amount)?

      YesNo

    • If the answer to (j) is yes, define each subgroup.
    • Are there other specialized subgroups in your target population about which you need specific data?

      YesNo

    • If the answer to (l) is yes, list and describe each specialized subgroup (e.g., elderly, adolescents, females, drug users, gang members, violent offenders).
    IV. Stakeholders

    List all who have some responsibility for commissioning a needs assessment, whether internal or external to your organization. Place them in order of decision-making influence from the most influential in commissioning and directing the implementation of a needs assessment to the least influential, and list their names and relationship to the organization needing the assessment. Indicate also the general type of assessment they expect (quantitative or qualitative) and list the reason for their preference.

    Influential persons are those who need to be satisfied and from whom positive or negative consequences may come about (e.g., having to redo the needs assessment or receiving acclaim for it; generating praise or criticism of the implementing organization; experiencing an increase, maintainance, or reduction in the agency's funds).

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    • Name Relationship to Your Organization

      Type of needs assessment desired or expected by this person:

      • Quantitative
      • Qualitative
      • Mixed
      • Doesn't care.
      • Not sure (Please find out.)

      Indicate reason for choice and list other requirements:

    V. Resource Availability
    • List the amount of funds you have to conduct a needs assessment:
    • List the people who will help conduct the needs assessment and indicate if they have training or experience in conducting needs assessments.
    • List any language capabilities they possess that can be used to interact with study participants who may not have a command of English:
    • List cultural training or experience these persons hold that may be used for collecting data for the needs assessment:
    • Check off the equipment and supplies you have access to (check all that apply):
      • Printing equipment (e.g., photocopy machine)
      • Computer with statistical analysis program
      • Computer with word processing software
      • Postage
      • Transportation—vehicle or reimbursement for car
      • Work space and storage space for study materials
      • Other equipment or supplies:
    • How much time, in months, do you have to undertake a needs assessment from start to finish?
    • Is the amount of time in (f) inflexible?

      YesNoNot sure (Please find out.)

    VI. Use of the Information
    • Indicate the expected uses of the results of the study. Check off all that apply. Results from the needs assessment will enable you to:
      • Alter organization functions and services as appropriate.
      • Fulfill a requirement by your sponsors.
      • Fulfill another externally imposed requirement:
      • Support existing views and perceptions.
      • Justify the existence of your organization.
      • Generate additional funding.
      • Other:
    • Are two needs assessments being planned for a pre- or postevaluation study?

      YesNo

    • How do you plan to report the results of the needs assessment? Select one:
      • A comprehensive, formal written report
      • A brief, formal written report
      • A short, informal written report
      • Only a verbal presentation
      • Other:
    • Who will be responsible for documenting the results of the study?
    VII. Decision-Making Guide

    Place a check mark next to the most appropriate response to each question below. Use your previous responses to help you respond to these questions.

    • Type of needs assessment:
      • Quantitative
      • Qualitative
      • Mixed
      • Other:
      • Not sure. Explain:
    • Type of questionnaire or data source:
      • Secondary data
      • Interview
      • Key informant
      • Focus group
      • Survey
      • Other:
      • Not sure. Explain:
    • Form of questions to ask:
      • Open-ended
      • Structured (fixed choice)
      • Semistructured
      • Other:
      • Not sure. Explain:
    • Will your sample be stratified (i.e., participants selected based on such demographic characteristics as gender, ethnicity, income, education)?

      YesNoNot sure

      Explain:

    • If the answer to (d) is yes, what are your stratifying variables?
      • Gender/sex
      • Ethnicity
      • Income
      • Education
      • Age
      • Other:
      • Other:
      • Other:
    • How many participants do you believe you need?
    • If selecting a stratified sample, how many participants will you need in each stratum (e.g., gender: 30 men and 30 women)?

      Stratum (Variable)Number in Each Stratum Needed

    • Participant selection method to be used:
      • Based on participant's expertise (focus/key informant)
      • Quota sampling method
      • Interval
      • Judgment
      • Snowball
      • Systematic
      • Other:
      • Not sure. Explain:
    • What types of demographic variables will you include in your questionnaire (e.g., gender, ethnicity)?

      If none, state why:

    • Will you conduct a pretest?

      YesNoNot sure

      If no or not sure, explain why:

      If yes, how many participants will you need?

    • Select the type of statistical analysis you will use:
      • Descriptive
      • Inferential
      • Mixed
      • Other:
    VIII. Sociopolitical and Economic Realities

    Describe in detail the population of focus–its social, cultural, and economic diversity.

    Mindful of the population of focus, list any controversies in the community, or among critical stakeholders that are likely to reduce the importance, use or impact of the findings from a needs assessment.

    List specific sources of funding that are realistically available to either support existing programs and/or services if a needs assessment demonstrates need.

    IX. General Checklist

    Place a check mark next to each section title below that you have addressed satisfactorily. Also indicate other concerns that need to be satisfied before the needs assessment is conducted.

    • I. Purpose and Objectives
    • II. Roles and Responsibilities
    • III. Targeted Population and Subgroups
    • IV. Stakeholders
    • V. Resource Availability
    • VI. Use of the Information
    • VII. Decision Making Guide
    • VIII. Sociopolitical and Economic Realities
    • Other:
    • Other:
    • Other:

    Appendix B: Pan-Acculturation Scale (PAN)

    —English Version—

    (Spanish Version Is Available)

    Fernando I. Soriano, PhD

    California State University–San Marcos Department of Human Development San Marcos, CA 92096

    Please Note:

    Please inform the author in writing of any planned use of the Pan-Acculturation Scale. Use of the PAN implies agreement to provide Dr. Soriano with a copy of your data set(s), which will be used solely for updating psychometric properties of the measures. Mail notifications of intended uses of the measure and completed data sets to the address listed above.

    Author holds copyright of scale. Used here with permission.

    Pan-Acculturation Scale Scoring Instructions

    The Pan-Acculturation Scale (PAN) measures acculturation. Acculturation is defined as a measure of the psychosocial and cultural participation of individuals within the American or dominant culture and one's “other culture” or culture of origin. Using the PAN, one can derive up to six scores measuring one's affinities to either or both American or one's other culture. Alternatively, a single acculturation measure can be derived, as well.

    The following are the cultural dimensions possible and their operational definitions.

    American Cultural Orientation Subscale

    One's affinity toward American culture is measured by calculating the proportion of “Yes” responses among valid responses for the “American Culture Scale” portion of the PAN scale (exclude nonresponse items in deriving the proportion).

    Other Culture Orientation Subscale

    One's affinity towards one's other culture is measured by calculating the proportion of “Yes” responses among valid responses for the “Other Culture Scale” portion of the PAN scale (exclude nonresponse items in deriving the proportion).

    Keep in mind that scores on both American Cultural and Other Cultural Orientation Scales measure the proportion of “yes” responses to each subscale and do not assume a particular affinity to either (i.e., that the score is indicative of one's “Americanness” or “other-cultureness”).

    Acculturation Typologies

    One can derive up to four typologies based on responses to the PAN. These are (1) American Oriented, (2) Other Non–American Culture Oriented, (3) Bicultural (oriented in both about equally), and (4) Marginalized (oriented in neither).

    Operationally Defining Types

    American Oriented. An American-Oriented person is one who responded “yes” to over 50% of the valid responses on the American Orientation Subscale.

    Other Non–American Culture Oriented. An Other Non–American culturally oriented person is one who responded “yes” to over 50% of the valid responses on the Other Culture Orientation Subscale.

    Bicultural. A biculturally oriented individual is one who met the orientation criteria for both American orientation and for non–American cultural orientation. That is, he or she marked “yes” to over 50% of the valid nonmissing items on both the American and the Other Culture Subscales.

    Marginal. A marginalized person is one who is oriented toward neither American culture nor toward their Other Culture. That is, he or she failed to marked “yes” to over 50% of the valid nonmissing items on both the American and the Other Culture Subscales.

    Single Measure of Acculturation

    A single measure of acculturation can be derived by recoding responses so that “1” = My Cultural Group response option, “2” = Both response option, and “3” = American Culture response option. The Neither response option would be coded as “missing data.” Few respondents mark this response, so the number of missing responses should not be great and therefore affect the stability or reliability of the measure. After recoding responses as indicated, the total sum of valid responses (minus the “Neither” response) would be the single acculturation measure.

    Importance of Cultural Groups (PAN) English

    Everyone belongs to one cultural or ethnic group. Examples of cultural groups include Mexican American, Irish, German, Chinese, and African American, among others. Some people are a mixture of several cultural groups. When this is true, a person might find one cultural group more influential than another. Cultural and ethnic groups are important because they can influence our beliefs and traditions and how we think, feel, and act. These questions are about your ethnicity or your cultural group and how you feel about it or react to it. What cultural group is important to you besides the American Cultural Group?

    My Important Cultural Group (Besides American) Is:_____

    I will read to you a series of statements. Please tell me whether each of your own personal, individual characteristics listed below is like the cultural group you just wrote or just told me, or is more like American Culture, both cultures, or neither culture. Pick only one response for each item.

    * Use this item with bilingual spans speaking respondents only.

    References

    Altschuld, J. W., & Witkin, B. R. (2000). From needs assessment to action: Transforming needs into solution strategies. Thousand Oaks, CA: Sage.
    Baker, B. O., Hardyck, C. D., & Petrinovich, L. F. (1966). Weak measurements vs. strong statistics: An empirical critique of S. S. Stevens' proscriptions on statistics. Educational and Psychological Measurement, 26, 291–309. http://dx.doi.org/10.1177/001316446602600204
    Barrera, M. R., Toobert, D., Strycker, L., & Osuna, D. (2012). Effects of acculturation on a culturally adapted diabetes intervention for Latinas. Health Psychology, 31(1), 51–54. http://dx.doi.org/10.1037/a0025205
    Bazeley, P. (2007). Qualitative data analysis with NVivo. Thousand Oaks, CA: Sage.
    Beatty, D. L., Kamarck, T. W., Matthews, K. A., & Shiffman, S. (2011). Childhood socioeconomic status is associated with psychosocial resources in African Americans: The Pittsburgh Healthy Heart Project. Health Psychology, 30(4), 472–480. http://dx.doi.org/10.1037/a0024304
    Berger, R. M., & Patchner, M. A. (1988a). Implementing the research plan: A guide for the helping professions. Newbury Park, CA: Sage.
    Berger, R. M., & Patchner, M. A. (1988b). Planning for research: A guide for the helping professions. Newbury Park, CA: Sage.
    Blythe, B. J., & Tripodi, T. (1989). Measurement in direct practice. Newbury Park, CA: Sage.
    Calderón, J. L., Morales, L. S., Liu, H., & Hays, R. D. (2006). Variation in the readability of items within surveys. American Journal of Medical Quality, 21(1), 49–56.
    Campaign for Youth Justice. (2007). Jailing juveniles: The dangers of incarcerating youth in adult jails in America. Washington, DC: Author.
    Carey, M. P., Senn, T. E., Vanable, P. A., Coury-Doniger, P., & Urban, M. A. (2010). Brief and intensive behavioral interventions to promote sexual risk reduction among STD clinic patients: Results from a randomized controlled trial. AIDS Behavior, 14, 504–517. http://dx.doi.org/10.1007/s10461-009-9587-1
    CDC (Centers for Disease Control and Prevention). (2008). Community Health and Program Services (CHAPS): Health disparities among racial/ethnic populations. Atlanta, GA: US Department of Health and Human Services.
    CDC (Centers for Disease Control and Prevention). (2011). Adolescent and school health: Health disparities. Retrieved from http://www.cdc.gov/healthyyouth/disparities/index.htm
    CDC NCHS (Centers for Disease Control and Prevention National Center for Health Statistics). (2011, February 23). Draft 2011 NHIS questionnaire–Household composition. Retrieved from ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Survey_Questionnaires/NHIS/2011/english/qhhld.pdf
    Chaloupka, F. J. (1999). The economic analysis of substance use and abuse: An integration of econometric and behavioral economic research. Chicago, IL: University of Chicago Press. http://dx.doi.org/10.7208/chicago/9780226100494.001.0001
    Chavez, L. R. (1986). Mexican immigration and health care: A political economy perspective. Human Organization, 45, 344–352. http://dx.doi.org/10.17730/humo.45.4.r4l7250376270092
    Cohen, J. (1988). Statistical power analysis for the behavioral sciences (
    2nd ed.
    ). Hillsdale, NJ: Lawrence Erlbaum.
    Coley, S. M., & Scheinberg, C. A. (1990). Proposal writing. Newbury Park, CA: Sage.
    Cook, M. B., McGlynn, K. A., Devesa, S. S., Freedman, N. D., & Anderson, W. F. (2011). Sex disparities in cancer mortality and survival. Cancer Epidemiology Biomarkers Prevention, 20(8), 1629–1637. http://dx.doi.org/10.1158/1055-9965.EPI-11-0246
    Corrigan, P. W., & O'Shaughnessy, J. R. (2007). Changing mental illness stigma as it exists in the real world. Australian Psychologist, 42(2), 90–97. http://dx.doi.org/10.1080/00050060701280573
    Cuellar, I., Arnold, B., & Maldonado, R. (1995). Acculturation Rating Scale for Mexican Americans–II: A revision of the original ARSMA scale. Hispanic Journal of Behavioral Science, 17, 275–304. http://dx.doi.org/10.1177/07399863950173001
    Currin, J. B., Hayslip, B. R., & Temple, J. R. (2011). The relationship between age, gender, historical change, and adults' perceptions of mental health and mental health services. International Journal of Aging & Human Development, 72(4), 317–341. http://dx.doi.org/10.2190/AG.72.4.c
    CWDA-CSAC (County Welfare Directors Association of California and California State Association of Counties). (2009). Human services in a time of economic crisis: An examination of California's safety-net programs and related economic benefits for communities. Sacramento: County Welfare Directors Association of California and California State Association of Counties. Retrieved from http://www.cwda.org/uploads/CSAC-CWDA-Joint-Repor0409.pdf
    de Vaus, D. A. (1990). Surveys in social research (
    2nd ed.
    ). Boston, MA: Allen & Unwin.
    Duncan, G. J., & Petersen, E. (2001). The long and short of asking questions about income, wealth and labor supply. Social Science Research, 30(2), 248–263. http://dx.doi.org/10.1006/ssre.2000.0696
    Friedman, D. B., & Hoffman-Goetz, L. (2006). A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Education and Behavior, 33(3), 352–373. http://dx.doi.org/10.1177/1090198105277329
    Galea, S., & Vlahov, D. (Eds.). (2005). Handbook of urban health: Populations, methods, and practice. New York, NY: Springer. http://dx.doi.org/10.1007/b104167
    Gilliam, G. A., Eke, A., Aymer, F., & O'Neil, C. (2001). Developing a theory-based, culturally sensitive intervention for adolescents: The Antigua School Project. Journal of HIV/AIDS Prevention and Education for Adolescents and Children, 4(2–3), 105–126.
    Gilmore, G. D., & Campbell, M. D. (2005). Needs capacity assessment strategies for health education and health promotion. Sudbury, MA: Jones and Bartlett.
    Hagger-Johnson, G., Mõttus, R., Craig, L. A., Starr, J. M., & Deary, I. J. (2012). Pathways from childhood intelligence and socio-economic status to late-life cardiovascular disease risk. Health Psychology. doi:10.1037/a0026775
    Howard, K. A. S, Carlstrom, A. H., Katz, A. D., Chew, A. Y., Ray, G. C., Laine, L., & Caulum, D. (2011). Career aspiration of youth: Untangling race/ethnicity, SES and gender. Journal of Vocational Behavior, 79(1), 98–109. http://dx.doi.org/10.1016/j.jvb.2010.12.002
    Institute of Medicine of the National Academies. (2006). Effect of the HIPAA Privacy Rule on health research: Proceedings of a workshop presented to the National Cancer Policy Forum. Washington, DC: The National Academies Press. Retrieved from http://www.iom.edu/Reports/2006/Effect-of-the-HIPAA-Privacy-Rule-on-Health-Research--Proceedings-of-a-Workshop-Presented-to-the-National-Cancer-Policy-Forum.aspx
    James, D., & Glaze, L. (2006). Mental health problems of prison and jail inmates (NCJ 213600). Washington, DC: Bureau of Justice Statistics. http://dx.doi.org/10.1037/e557002006-001
    Johnson, D. E., Meiller, L. R., Miller, L. C., & Summers, G. R (1987). Needs assessment: Theory and methods. Ames: Iowa State University Press.
    Johnston, L. D., O'Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2011). Monitoring the Future national results on adolescent drug use: Overview of key findings, 2010. Ann Arbor: The University of Michigan, Institute for Social Research.
    Jose-Kampfner, C. (1995). Post-traumatic stress reactions in children of imprisoned mothers. In K.Gabel & D.Johnston (Eds.), Children of incarcerated parents (pp. 89–100). New York, NY: Lexington Books.
    Kalton, G. (1983). Introduction to survey sampling. Newbury Park, CA: Sage.
    Karriker-Jaffe, K. J. (2011). Areas of disadvantage: A systematic review of effects of area-level socioeconomic status on substance use outcomes. Drug and Alcohol Review, 30(1), 84–95. http://dx.doi.org/10.1111/j.1465-3362.2010.00191.x
    Kaufman, R., & English, F. (1979). Needs assessment: Concept and application. Englewood Cliffs, NJ: Educational Technology.
    Kershaw, K. N., Mezuk, B., Abdou, C. M., Rafferty, J. A., & Jackson, J. S. (2010). Socioeconomic position, health behaviors, and C-reactive protein: A moderated-mediation analysis. Health Psychology, 29(3), 307–316. http://dx.doi.org/10.1037/a0019286
    Kettner, P. M., Moroney, R. M., & Martin, L. L. (1990). Designing and managing programs: An effectiveness-based approach. Newbury Park, CA: Sage.
    Kim, B. S. K. (2007). Adherence to Asian and European American cultural values and attitudes toward seeking professional psychological help among Asian American college students. Journal of Counseling Psychology, 54, 474–480. http://dx.doi.org/10.1037/0022-0167.54.4.474
    Kirsch, I. S., Jungeblut, A., Jenkins, L., & Kolstad, A. (2002). Adult literacy in America: A first look at the findings of the National Adult Literacy Survey (
    3rd ed.
    ). Washington, DC: US National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs93/93275.pdf
    Kruegar, R. A. (1998). Analyzing and reporting focus group results (Focus Group Kit 6). Thousand Oaks, CA: Sage.
    Krueger, R. A. (2000). Focus groups: A practical guide for applied research. Thousand Oaks, CA: Sage.
    Lauffer, A. (1982). Assessment tools: For practitioners, managers, and trainers. Beverly Hills, CA: Sage.
    Leech, N. L., & Onwuegbuzie, A. J. (2011). Beyond constant comparison qualitative data analysis: Using NVivo. School Psychology Quarterly, 26(1), 70–84. http://dx.doi.org/10.1037/a0022711
    Levy, J. J. (2004). Review of Acculturation: Advances in theory, measurement, and applied research. Counselling Psychology Quarterly, 17(1), 135–136.
    Lewins, A., & Silver, C. (2007). Using software in qualitative research: A step-by-step guide. Thousand Oaks, CA: Sage.
    Loue, S., & Quill, B. E. (Eds). (2001). The handbook of rural health. New York, NY: Kluwer Academic. http://dx.doi.org/10.1007/978-1-4757-3310-5
    McDavid, J. C., & Hawthorn, L. R. L. (2006). Program evaluation and performance measurement: An introduction to practice. Thousand Oaks, CA: Sage.
    McGowan, A., Hahn, R., Liberman, A., Crosby, A., Fullilove, M., Johnson, R.Stone, G. (2007). Effects on violence of laws and policies facilitating the transfer of juveniles from the juvenile justice system to the adult justice system: A systematic review. American Journal of Preventative Medicine, 32(4S), S7-S28. http://dx.doi.org/10.1016/j.amepre.2006.12.003
    Mendoza, R. H. (1989). An empirical scale to measure type and degree of acculturation in Mexican-American adolescents and adults. Journal of Cross-Cultural Psychology, 20(4), 372–385. http://dx.doi.org/10.1177/0022022189204003
    Minkler, M., & Hancock, T. (2008). Community-driven asset identification and issue selection. In M.Minkler & N.Wallerstein (Eds.), Community-based participatory research for health: From process to outcomes (
    2nd ed.
    , pp. 153–169). San Francisco, CA: Jossey-Bass.
    Moore, J. W. (1991). Going down to the barrio: Homeboys and homegirls in change. Philadelphia, PA: Temple University Press.
    Myers, B. J., Smarsh, T. M., Amlund-Hagen, K., & Kennon, S. (1999). Children of incarcerated mothers. Journal of Child and Family Studies, 8(1), 11–25. http://dx.doi.org/10.1023/A:1022990410036
    Neddenriep, C. E., & Hale, A. D. (2011). Introduction to the special issue: Reading comprehension: Assessment and intervention for understanding. Psychology in the Schools, 48(1), 1–3. http://dx.doi.org/10.1002/pits.20538
    Neuber, K. A. (with Atkins, W. T., Jacobson, J. A., & Reuteman, N. A.). (1980). Needs assessment: A model for community planning. Beverly Hills, CA: Sage.
    NIDA (National Institute on Drug Abuse). (2011). NIDA for teens: The science behind drug abuse: Marijuana. Retrieved from http://teens.drugabuse.gov/facts/facts_mj1.php
    NIH (National Institutes of Health). (2001). NIH policy and guidelines on the inclusion of women and minorities as subjects in clinical research–amended, October, 2001. Bethesda, MD: Office of Extramural Research, National Institutes of Health. Retrieved from http://grants.nih.gov/grants/funding/women_min/guidelines_amended_10_2001.htm
    Passel, J. S., & Cohn, D. (2008). Trends in unauthorized immigration: Undocumented inflow now trails legal inflow. Washington, DC: Pew Hispanic Center. Retrieved from http://www.pewhispanic.org/2008/10/02/trends-in-unauthorized-immigration/
    Paz, S. H., Liu, H., Fongwa, M. N., Morales, L. S., & Hays, R. D. (2009). Readability estimates for commonly used health-related quality of life surveys. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care & Rehabilitation, 18(7), 889–900. http://dx.doi.org/10.1007/s11136-009-9506-y
    Petersen, D. J., & Alexander, G. R. (2001). Needs assessment in public health: A practical guide for students and professionals. New York, NY: Springer.
    Posavac, E. J., & Carey, R. G. (2007). Program evaluation: Methods and case studies (
    7th ed.
    ). Upper Saddle River, NJ: Pearson Prentice Hall.
    QSR International. (2008). NVIVO: Version 8; Reference guide. Victoria, Australia: Doncaster.
    Reviere, R., Berkowitz, S., Carter, C. C., & Ferguson, C. G. (1996). Needs assessment: A creative and practical guide for social scientists. Washington, DC: Taylor & Francis.
    Riley, J. C., & Milbank Memorial Fund. (2008). Low income, social growth, and good health: A history of twelve countries. Berkeley: University of California Press.
    Royse, D., Staton-Tindall, M., Badger, K., & Webster, J. M. (2009). Needs assessment. New York, NY: Oxford University Press. http://dx.doi.org/10.1093/acprof:royes/9780195368789.001.0001
    Ruffin, J. (2011, May 11). Fiscal year 2012 budget request: Witness appearing before the Senate Subcommittee on Labor-HHS-Education Appropriations: Department of Health and Human Services, National Institutes of Health. Retrieved from http://www.nimhd.nih.gov/about_ncmhd/Congressional%20Testimony%20%205-11-11.pdf
    Saad, C. S., Sue, S., Zane, N., & Cho, Y., II (2012). The relationship between education and ethnic minority factors in income among Asian Americans. Asian American Journal of Psychology. doi:10.1037/a0026867.
    Salisbury, E. J., Van Voorhis, P., & Spiropoulos, G. V. (2009). The predictive validity of a gender-responsive needs assessment: An exploratory study. Crime & Delinquency, 55(4), 550–585. http://dx.doi.org/10.1177/0011128707308102
    Sam, D. L., & Berry, J. W. (2010). Acculturation: When individuals and groups of different cultural backgrounds meet. Perspectives on Psychological Science, 5(4), 472–481. http://dx.doi.org/10.1177/1745691610373075
    Satcher, D. (2005). What if we were equal? A comparison of Black-White mortality gap in 1960 and 2000. Health Affairs24(2), 459–464. http://dx.doi.org/10.1377/hlthaff.24.2.459
    Schaefer, M. (1987). Implementing change in service programs: Project planning and management. Newbury Park, CA: Sage.
    Schirmer, S., Nellis, A., & Mauer, M. (2009). Incarcerated parents and their children: Trends 1991-2007. Washington, DC: The Sentencing Project. Retrieved from http://www.sentencingproject.org/doc/publications/publications/inc_incarceratedparents.pdf
    Soriano, F. I., Ho, J., Yeh, M., McCabe, K., & Hough, R. (2012). PAN Acculturation Scale: A psychometric analysis. Unpublished report. Child and Adolescent Services Research Center, Rady's Children's Hospital, San Diego, CA.
    Stephenson, M. (2000). Development and validation of the Stephenson Multigroup Acculturation Scale (SMAS). Psychological Assessment, 12, 77–88. http://dx.doi.org/10.1037/1040-3590.12.1.77
    Stewart, D. W., Shamdasani, P. N., & Rook, D. W. (2007). Focus groups: Theory and practice. Thousand Oaks, CA: Sage.
    Suinn, R. M. (2010). Reviewing acculturation and Asian Americans: How acculturation affects health, adjustment, school achievement, and counseling. Asian American Journal of Psychology, 1(1), 5–17. http://dx.doi.org/10.1037/a0018798
    Suinn, R. M., Ahuna, C., & Khoo, G. (1992). The Suinn-Lew Asian Self-Identity Acculturation Scale: Concurrent and factorial validation. Educational and Psychological Measurement, 52(4), 1041–1046. http://dx.doi.org/10.1177/0013164492052004028
    Sullivan, M. M., & Rehm, R. (2005). Mental health of undocumented Mexican immigrants: A review of the literature. Advances in Nursing Science, 28(3), 240–251. http://dx.doi.org/10.1097/00012272-200507000-00006
    Thomas, L. R., Donovan, D. M., & Sigo, R. L. W. (2010). Identifying community needs and resources in a Native community: A research partnership in the Pacific Northwest. International Journal of Mental Health and Addiction, 8(2), 362–373. http://dx.doi.org/10.1007/s11469-009-9233-1
    US Census Bureau. (2011a, March 24). 2010 census shows America's diversity [press release]. Retrieved from http://www.census.gov/newsroom/releases/archives/2010_census/cb11-cn125.html
    US Census Bureau. (2011b, September 13). Income, poverty and health insurance coverage in the United States: 2010 [press release]. Retrieved from http://www.census.gov/newsroom/releases/archives/income_wealth/cb11-157.html
    US Census Bureau. (2012). American Community Survey 2012 questionnaire. Retrieved from http://www.census.gov/acs/www/methodology/questionnaire_archive/
    USHUD (US Department of Housing and Urban Development). (2010). The 2010 annual homeless assessment report to Congress. Washington, DC: US Department of Housing and Urban Development, Office of Community Planning and Development. Retrieved from http://www.hudhre.info/documents/2010HomelessAssessmentReport.pdf
    Van de Poel, E., O'Donnell, O., & Van Doorslaer, E. (2007). Are urban children really healthier? Evidence from 47 countries. Social Science and Medicine, 65(10), 1986–2003. http://dx.doi.org/10.1016/j.socscimed.2007.06.032
    Vondras, D. D., Powless, M. R., Olson, A. K., Wheeler, D., & Snudden, A. L. (2005). Differential effects of everyday stress on the episodic memory test performances of youth, mid-life, and older adults. Aging and Mental Health, 9(1), 60–70. http://dx.doi.org/10.1080/13607860412331323782
    Walters, N. P., & Trevelyan, E. N. (2011, November). The newly arrived foreign-born population of the United States: 2010 (American Community Survey Briefs, ACBR/10-16). Washington, DC: US Census Bureau. Retrieved from http://www.census.gov/prod/2011pubs/acsbr10-16.pdf
    Witkin, B. R. (1984). Assessing needs in educational and social programs. San Francisco, CA: Jossey-Bass.
    Wolfer, L. (2007) Real research: Conducting and evaluating research in the social sciences. Boston, MA: Allyn & Bacon.
    Woolf, S. H., Johnson, R. E., Fryer, G. E., Rust, G., & Satcher, D. (2004). The health impact of resolving racial disparities: An analysis of US mortality data. American Journal of Public Health, 94(12), 2078–2081. http://dx.doi.org/10.2105/AJPH.94.12.2078
    Yoon, E., Langrehr, K., & Ong, L. Z. (2011). Content analysis of acculturation research in counseling and counseling psychology: A 22-year review. Journal of Counseling Psychology, 58(1), 83–96. http://dx.doi.org/10.1037/a0021128

    About the Author

    Fernando I. Soriano is Professor of Human Development at California State University–San Marcos. An applied researcher and academician, he has written numerous publications focusing on such social problems as gang membership, youth violence, delinquency, substance abuse, and AIDS. He has sat on several national committees, including the Commission on Violence and Youth for the American Psychological Association, and assisted with the US Surgeon General's report on youth violence. A specialty of his is understanding and measuring the importance of culture and cultural factors in youth violence. He has been involved in community program development and evaluation for several years.


    • Loading...
Back to Top

Copy and paste the following HTML into your website