Introduction

The Copenhagen Psychosocial Questionnaire (COPSOQ) was developed by the Danish National Research Centre for the Working Environment to be a valid instrument for the assessment of the psychosocial work environment and for facilitating communication between workplaces, work environment professionals, and researchers (Kristensen et al., 2005; Pejtersen et al., 2010). It is a comprehensive, generic instrument including numerous dimensions based on an eclectic set of theories on psychosocial factors at work and on empirical research, rather than being linked to any one particular theory. The second version of the instrument, COPSOQ II, exists in a short, a medium and a long version.

The COPSOQ II reflects a broad perspective on both positive and negative aspects of the psychosocial work environment: Demands at work, work organization and job contents, interpersonal relations and leadership, work-family conflict, health, well-being, and offensive behaviours as well as social capital at workplace level (Pejtersen et al., 2010).

Today, COPSOQ versions I & II have been translated into more than 25 languages, and research from many countries has contributed to the validation of its use in research as well as in practical psychosocial interventions in organizations (Kristensen, 2010; Nübling et al., 2014). Published validation studies of COPSOQ II have primarily been based on psychometrics and statistical analyses such as evaluations of the internal consistency, explorative analyses of the factor structure, ceiling and floor effects, and assessment of construct and criterion validity (e.g. Aust et al., 2007; Bjorner and Pejtersen, 2010; Moncada et al., 2014; Pejtersen, Bjorner, and Hasle, 2010), or test-retest stability (Rosário, Fonseca, and da Costa, 2014). In contrast, only sparse information exists concerning the evaluation of the content validity of the scales – and neglecting this part of validation concerns most other similar instruments. This analysis is, however, of the utmost importance for any instrument aimed at measuring working conditions in a changing work life, where complex issues such as interpersonal relationships need to be addressed in addition to the traditional focus on more concrete factors such as job factors.

Content validity can be understood as how well an instrument reflects the construct we want to measure and thereby it is central for the inferences that can be made from using the instrument (Mokkink et al., 2010; Streiner, Norman, and Cairney, 2014). Traditionally, content validation includes expert assessments of relevance and coverage of an instrument based on a definition of the construct. Based on quantitative analyses of agreement and results from pilot studies, the instrument is subsequently constructed (Streiner, Norman, and Cairney, 2014). The COPSOQ scales were constructed based on theoretical and statistical reasoning, and the frequent applications in research as well as at workplaces suggest that its relevance and coverage is widely accepted. While most COPSOQ items are rather straightforward asking about personal perceptions or concrete aspects of working conditions, more complex aspects of work life are also addressed. Workplace social capital, which both researchers and practitioners show high interest in (Kristensen, 2010), is a group phenomenon of high complexity. Therefore, a better understanding of the validity of the measurement is needed. In this study, our aim is therefore to particularly focus on the meanings attributed to the social capital items included in the COPSOQ II.

Social capital in COPSOQ II

Social capital was introduced as a concept by Bourdieu (1986) and is considered to have roots dating back to Durkheim’s research indicating the importance of social cohesion for well-being in groups (Kawachi and Berkman, 2000). Social capital can be operationalized in different ways, and this also applies to workplace social capital, which during the last two decades has emerged as a research field of increasing interest. Putnam introduced a division of social capital into bonding (relations bonding people in a group together) and bridging social capital (relations bridging groups) (Putnam, Leonardi, and Nanetti, 1994). A few years later the linking dimension of social capital was suggested (Szreter and Woolcock, 2004). A recently published concept analysis shows that the research done on workplace social capital among nurses points to a focus on networks of social relationships at work, shared assets and shared ways of knowing and being (Read, 2014). This also applies to the frequently used Finnish measure for workplace social capital, which asks respondents directly about their perception of cohesion, trust, and knowledge sharing in the group, and how the employees perceive they are treated by their superior (Kouvonen et al., 2006).

Interpersonal relationships characterized by mutual trust and justice are considered fundamental pillars of workplace social capital (Healy and Côté, 2001; Kawachi et al., 2013; Olesen et al., 2008). In line with this, new scales for trust and justice were introduced in COPSOQ II and the intention behind these was formulated like this: “The purpose of these items is to get a picture of the whole workplace (company) of the respondent and not just the person’s own job or department” (National Research Centre for the Working Environment, 2007; Kristensen, 2010; Pejtersen et al., 2010). The scales for trust and justice constitute a core in the measurement of workplace social capital in the COPSOQ instrument (Kiss et al., 2014; Kristensen, 2010; Lundstrøm et al., 2014; Olesen et al., 2008). COPSOQ and the Finnish measure share focus by addressing bonding social capital through trust and openness, but broaden the linking aspect of social capital from a top-down perspective to a mutuality in the relations between superior and staff and even organizational justice is included.

Workplace social capital is important for employees as well as for the organizations as it has been shown to impact, among other things, job satisfaction and engagement in quality improvements at work in addition to health outcomes (Oksanen et al., 2013; Smith et al., 2013; Strömgren et al., 2015). In addition, in previous studies social capital has been related to organizational efficiency based on, e.g., work engagement and organizational commitment (Hakanen, Perhoniemi and Rodríguez Sánchez, 2012).

Workplace social capital does not concern individual attributes or perceptions, but addresses the shared beliefs and values coming from relations at work. Therefore, while most COPSOQ II items ask for respondents’ personal experiences and perceptions of the working environment, the items included in the scales for workplace social capital ask the individual respondent about group-level phenomena. Organizational justice, for example, is measured by asking the individual to answer in relation to the workplace as a whole rather than referring to the personal work situation, in other words with a shift in referent from the individual to the collective. This way of composing a group-level construct based on individual-level survey data, a referent-shift consensus model (Chan, 1998; Van Mierlo, Vermunt and Rutte, 2009), is considered to be an appropriate way of measuring organizational culture and climate (Glisson and James, 2002). A shift of referent is required as a consequence of the theoretical understanding of social capital as a group phenomenon which distinguishes COPSOQ from other measures of workplace social capital. The shift of referent in COPSOQ II is underlined by the use of words like employees, workplace, and management in the questions and in the introductory text to this particular section of the questionnaire: “The next questions are not about your own job but about the workplace as a whole”.

Aims of the study

According to Tourangeau, respondents to questionnaires typically go through these four steps: 1) interpretation of the question, 2) retrieval of information and beliefs in order to answer the question, 3) judgement, and 4) reporting of the answer (Tourangeau, 1984; Tourangeau and Rasinski, 1988). In many cases some of these steps are gone through subconsciously, but when you ask a person to abstract from her/himself and instead look at the entire workplace it is likely that one or more of the aforementioned steps are more complicated. This in turn may result in reduced reliability and validity of the measures.

As one of the first steps in the ongoing validation study of the Swedish version of COPSOQ II, cognitive interviews were used (Berthelsen, Westerlund, and Kristensen, 2014; Berthelsen et al., 2014). The primary aims of the interviews were a) to identify potential problems in the questionnaire; b) to clarify how different concepts and questions were interpreted by the respondents, and thus c) to revise the Swedish version of COPSOQ II before further use of the instrument (Berthelsen, Westerlund, and Kristensen, 2014; Berthelsen et al., 2014).

The interview data suggested that particularly understanding of the social capital items, which are arguably somewhat more complex than other items in the questionnaire, deserved deeper analysis. Many companies are using COPSOQ II for workplace surveys and consider social capital as an appealing concept in addition to the more traditional focus on risk factors in the working environment. Thereby, this study can be of value for future developments of research questions based on a shift of referent for assessment of group construct, as well as for the large number of workplaces using the instrument for organizational development.

Cognitive interviewing covers different techniques which can be used for the improvement of questionnaire design and for achieving knowledge of how questions are understood in the target population. The basis is a presentation of survey questions to informants, followed by the collection of relevant data from the process of responding. Such information can be obtained through what in the literature is considered as two different approaches: a think-aloud procedure and probing (Beatty and Willis, 2007; Willis, 2005). However, these methods go well hand in hand as to some extent they have different advantages and drawbacks, and they are often combined (Beatty and Willis, 2007; Boeije and Willis, 2013; Willis, 2005).

The purpose of the present study is to focus on the meanings of trust and justice items included in the COPSOQ II instrument, and particularly: 1) to evaluate whether the intended shift of referent from the individual perspective to workplace perspective occurred in practice; 2) to analyse how response options were used; 3) to identify situations or circumstances that might make answering difficult; and 4) to achieve insight into how the questions were interpreted among the interviewees.

Methods

Participants

The interview study was designed to capture the perceptions of the questionnaire in the working population in Sweden and with special focus on health care employees (Berthelsen, Westerlund, and Kristensen, 2014; Berthelsen et al., 2014). The first and the fourth author’s network provided information on contact persons from different parts of Sweden who gave further suggestions of friends and colleagues ready to participate. Informants were selected from these lists to achieve variation in gender, age, region of residence, and occupation. Twenty female and six male informants participated covering age groups from under 30 to over 60 years of age. As the actual use and understanding of the Swedish language may differ among people coming from different parts of the country a geographical diversity was also striven for. Informants came from Umeå in the north to Malmö in the south, from Gothenburg in the west to Stockholm in the east of Sweden, and even informants with a non-native Swedish background participated.

The work experience of the informants ranged from 1 to 38 years, and their contractual hours from part-time to full-time (26–40 hours a week). People working in the private as well as the public sector were interviewed. From the health care sector, nurse’s aides, nurse’s assistants, nurses, specialized nurses and doctors participated, some of whom had managerial responsibilities. In a similar way all echelons within dentistry were selected. In addition, nine interviews were conducted with people working in sectors other than health care. This was done in order to achieve more variation and thereby better external validity of the results. These comprised traditional blue-collar workers, as well as e.g. specialists and project managers and came from industrial as well as creative settings.

Data collection

Interview data were collected in the autumn of 2013 by the fourth author and interviews were carried out to obtain information until saturation was achieved. Face-to-face interviews were conducted with concurrent probing. The probing consisted of fixed probes in addition to a more flexible form, following up on threads coming from the reflections of the informant.

Times and places for the interviews were decided in collaboration with the informants. The interviews lasted 40–70 minutes. Some informants preferred to be interviewed at home, while others chose a café or an undisturbed room at their workplace. The process of interviewing was continued until satisfactory formulations of questions and saturation of reflections, in relation to key concepts and content of the questions, were achieved. The interview method was based on a previously published interview guide (Berthelsen, Westerlund, and Kristensen, 2014).

Before an interview was started the informant was introduced to thinking aloud when answering questions as “how many windows do you have where you live?” The interview was started when the informant felt confident with the process. It was emphasized that the study purpose was to achieve knowledge about how respondents understood and reasoned in relation to the questions, rather than gaining knowledge about their concrete psychosocial working environment. Therefore, the informants were asked to keep their questionnaires and answers. The ethical consent for using the think-aloud procedure included people exclusively working in health care. Therefore, a revised interview protocol excluding thinking aloud was designed for informants from other kinds of jobs.

The questionnaire was given to the informants page by page. Most informants chose to read the text of each item aloud and to discuss their answer with themselves. Now and then, informants preferred to be silent. In such situations, the interviewer used open probing questions like “What did you think of when filling in this question?” or “How did you arrive at your answer on this question?” Thereby, probing was concurrent and flexible, aiming to stimulate the informant to elaborate further on their thoughts in relation to the questions and response alternatives. Standardized as well as unstandardized probes, initiated by the interviewer or triggered by informant behaviour, were used as suggested by Willis (2005).

Analyses

Interviews were conducted in five rounds. In addition to audio recording, short notes stating the interviewer’s general impressions and immediate reflections were taken after each interview. After each round the first and the fourth author reviewed the overall problems discovered during the interviews and made suggestions to the research group about changes in the questionnaire prompted by the findings. The final wording of the questionnaire has previously been published in Swedish (Berthelsen et al., 2014).

A systematic, directed content analysis (Hsieh and Shannon, 2005) of all quotations covering the domain Workplace social capital (Mutual trust between employees, Trust regarding management, and Justice) was conducted for the purpose of the present paper. A predetermined coding scheme was used to identify: 1) perspective (reflection on behalf of oneself only or abstraction to a broader perspective), 2) use of response options, 3) contexts challenging the process of answering and 4) overall reflections included in the retrieval and judgement processes leading to an answer for each item.

Results

The results will be presented theme by theme following the order of the research questions. Table 1 provides an overview of scales, items (including abbreviation of label) and an overview of the number of quotations in relation to shift of referent for each item. After quotations the abbreviation of the item label is referred to in parentheses.

Table 1

Overview of scales and items measuring workplace social capital and number of quotations in relation to shift of referent or problems answering.

Scale Item Number of quotations Item text Shift of referent Problems answering

Shift of referent No shift of referent No perspective identified Problem answering Tourangeau:
step 1: understand
step 2: retrieve
step 3: judge
step 4: report

Mutual trust between employees TE1 19 Do the employees withhold information from each other? 11 2 6 4 Understand: 2
Retrieve: 2
TE2 19 Do the employees withhold information from the management? 8 2 9 3 Retrieve: 3
TE3 20 Do the employees in general trust each other? 11 3 6 0
Trust regarding management TM1 21 Does the management trust the employees to do their work well? 6 2 13 0
TM2 20 Can you trust the information that comes from the management? 3 6 11 0
TM3 16 Does the management withhold important information from the employees? 4 4 8 5 Retrieve: 4
Report: 1
TM4 19 Are the employees able to express their views and feelings? 8 4 7 0
Justice JU1 22 Are conflicts resolved in a fair way? 9 2 11 7 Understand/retrieve: 1
Retrieve: 3
Retrieve/judge: 3
JU2 20 Are employees appreciated when they have done a good job? 11 5 4 1 Retrieve/judge: 1
JU3 22 Are all suggestions from employees treated seriously by the management? 5 9 8 3 Retrieve/report: 2
Unclear problem: 1
JU4 24 Is the work distributed fairly? 14 1 9 0

From individual to workplace perspective

The first aim was to evaluate whether the shift of referent from individual to workplace perspective actually occurred when informants answered the social capital items. In general, informants answered in accordance with the intended shift from individual to workplace perspective when answering, as exemplified here:

“… employees … probably, it’s my colleagues, I have to take into account here…. well … partly” (TM1)

A tendency was seen among informants to take their starting point in their personal opinions before abstracting their reflections to a broader perspective. An example is a quotation concerning the question “Do the employees withhold information from the management?”

“… well …no I don’t think so … often it is the boss who is the first informed if there is a problem, so to a very little extent” (TE2)

However, for one item the identification of perspective remained unclear (TM3) and problems in shifting referent were seen for two items (JU3 and TM2), illustrated by this quotation:

“Yes, I assume it is so, this is what I would have done if I had been the manager” (JU3)

In summary, for eight out of eleven items the majority of informants answered in a way which reflected the desired shift of referent.

Response options

The COPSOQ II response options for the items under study were created as a five-grade scale ranging from “to a very small extent” to “to a very large extent” and with a middle alternative: “somewhat”. The informants’ use of these response alternatives in different situations will be presented in the following section. Table 1 provides an overview of the frequency of problems with specific items in relation to the cognitive steps of the answering process.

The response alternatives with a clear direction were in general used as expected. Informants typically endorsed the response alternatives at the end of the scales when feeling sure about their answer. However, the use of the middle response option was more varied. Most often, the middle response option was used in accordance with the intention behind the option:

“Somewhat [Interviewer: what does somewhat mean in this case?] That sometimes they withhold information.” (TM3)

In contrast, later the same informant used the middle response alternative as a reply when not knowing what to answer:

“Well … actually I don’t know. Then it has to be somewhat, then … [interviewer: when you say don’t know, then you answer somewhat?] Mhm (confirming)” (JU1)

In general, when informants felt unsure about what to answer their typical reaction was either endorsing the middle response option or not answering at all, as in the following examples:

Actually, I don’t know, I write somewhat. (JU1)

Well … say so … I don’t know, I can’t answer it, I dare not answer. Well, I skip it. (TM3)

For the items JU1, TM3, TE1 and TE2 problems in answering the question were identified (JU1 and TM3: a third of the quotations, and TE1 and TE2: a fifth of the quotations, see Table 1). The main problem here was retrieving relevant information to answer the question (the second step in the process described by Tourangeau).

In summary, the informants used the response categories as expected and only the response option balancing in the middle covered different meanings. Retrieval of relevant information for answering constituted a problem in four out of eleven items.

Contextually challenging situations

Not surprisingly, the quotations showed that when an informant was rather new in a job it was perceived as difficult to answer some of the questions due to lack of knowledge about the phenomena the questions intended to cover. In addition, working simultaneously in different departments with unequal levels of trust, justice, and respect complicated finding an appropriate answer to the question.

“… At the one department yes, at the other no, so I answer: to a small extent”(JU4)

The word management also constituted some challenges in relation to what was actually included in the answer. However, the most common referent was the closest superior rather than the overall management.

“There is an ongoing discussion between us, the group leader often joins us and listens to what we say” (JU3)

In some situations the informant thought about a group of leaders, leaders at different organizational levels, or about several leaders, for example in the case of project or matrix organizations. In these situations it was difficult to answer because the concrete answer could differ depending on which leader and at which organizational level they were thinking about. In such cases informants tended to answer in relation to their closest superior.

“Oh … the management … it’s more like higher up … and I can’t answer for them, but our group leader: yes … so I put to a very large extent here.” (TM1)

In summary, we found three problems: uncertainty due to being newly employed; ambiguity due to a situation where the respondent works in several different workplaces; and lack of clarity in the question itself (what is meant by management).

Content of the scales

A further important aspect concerns the overall reflections included in the retrieval and judgement processes leading to an answer for each item. The aim here is to evaluate whether the content of what the respondents include in their answers corresponds with the intentions behind the scales.

Mutual trust between employees. According to the researchers behind the instrument “these three items are about the employees’ trust in each other and their behaviour in relation to management”, also called horizontal trust (National Research Centre for the Working Environment, 2007). As regards the items asking whether employees withhold information from each other or from the management (TE1, TE2) many informants focused on the word information and expressed doubts about what kind of information this could be. However, informants working in health care tended to express concerns about handling patient safety issues rather than aspects directly related to trust. In general, a tendency was to take into account the importance of information and whether withholding information was intentional or not.

The question asking whether employees in general trust each other (TE3) can be regarded as a core item of this scale as the reflections commonly concerned such issues as confidence and trust from the perspective of the group, as exemplified here:

“I don’t have any problems at all, but I think a little broader … oh … if I don’t base the answer on my own behalf, then I actually think it partly is so. I don’t believe that everybody has total confidence in …, I find that not everybody thinks the others do their job … like … ‘he is lazy and he doesn’t do etc.’ … so partly.” (TE3)

Trust regarding management. This scale is also intended to measure trust, but here the focus is on the relations between management and employees, so-called vertical trust (National Research Centre for the Working Environment, 2007). The question concerning whether management trusts the employees to do a good job (TM1) was in general answered in a straightforward way without much hesitation. The starting point for almost every informant was stating that they felt trusted and next, they provided examples on expressions of trust, taking in a broader perspective than their own. This item can thus be considered as a core item for the scale.

However, some informants who found their working environment less trustful took a small break and then reflected on examples illustrating signs of mistrust, often with offset in examples of feeling controlled.

“Yes, partly they do … there is a lot of …. in fact, they are controlling … there are reviews of patient records, they are looking after that everything is included as it should be, but then you actually also kind of control the employees, then you don’t really trust them, so partly I think it is” (TM1)

When it comes to trust in relation to information from the management (TM2) a predominant opinion was that as an employee one simply has to believe what is said, and that one is more likely not to receive any information or insufficient information than to be misled. On the other hand, the next item of the questionnaire actually asks if the management withholds information from the employees (TM3). However, this item was perceived as hypothetical and difficult to answer. In addition, a few informants expressed adverse reactions to this item, in particular informants in a managerial position, who tended to answer from a managerial perspective rather than regarding themselves as employees.

A typical reflection for the following item was that even though the employees felt able to express their views and feelings (TM4) that did not mean that the management listened actively.

“You may express yourself, but as to whether it leads further …” (TM4)

Workplace meetings and individual development meetings were typical examples of where such exchanges of views and feelings took place, and even the general atmosphere was referred to.

Justice. The majority of the informants answered the question whether conflicts were resolved in a fair way at their workplace (JU1) based on the extent of conflicts rather than on the way conflicts were actually resolved. Some informants described persistent conflicts, a situation which typically led to the answer to a very small extent when asked. In contrast, informants who found that their workplace was characterized by few conflicts typically expressed that it was hard to find an answer, which led them to choose the middle response option. In contrast, other informants in fact reflected on the concrete way conflicts were resolved. Those informants emphasized in particular their expectations of the leader and of the employees in relation to handling of conflicts at work.

Even though quite a few informants discussed working at non-appreciative workplaces, it did not seem to cause similar problems as in the JU1 to answer whether employees were appreciated when they had done a good job (JU2). The reflections on this item showed indications of a widespread culture characterized by little awareness of showing appreciation among managers and correspondingly low expectations of being appreciated among employees, as exemplified here:

“What demands can you actually have on being appreciated at your work? If you don’t hear anything, then you assume that you’re doing a good job or at least okay … partly” (4.1 JU 2)

Different kinds of manifestations of appreciation were stated. It could be extrinsic rewards such as salary or gifts from the management.

“No, you get the same salary whether you do a good job or a less good one” (4.3 JU2)

Still, there could also be intrinsic rewards coming, for example, from patients or management in the form of esteem, opportunities for development or simply of a feeling of being seen.

”Yes, I think so … [interviewer: what do you think about here?] Well, it’s after all a little bit about that they trust you … and then it’s also a little bit about which responsibilities you’re assigned … and when they say you’re doing a good job then it adds … well, you make use of my competences and then you grow and so …” (2.1 JU2)

Also the question whether all suggestions from employees were treated seriously by the management (JU3) seemed rather easy to answer and the reflections were largely grounded on concrete experiences and examples.

“Well, I don’t know … yesterday we were asked to place notes on a spruce tree telling what we can do better next year and then the manager would collect them to find out what is going to be done differently next year, so it feels like they care, to a very large extent” (JU3)

A frequent issue coming up was what should be understood by a fair way of distributing the work (JU4).

“Yes, but it depends actually on what you mean by a fair way. If you mean that everybody should do the same … but we are doing it from competences. And if I have the competence, the right competence for the task … then I believe to a very large extent” (JU4)

In some situations this question concerning fair distribution of the work was perceived as less relevant, especially in situations where the role of the management did not correspond well with what was asked.

“Well – is distributed in a fair way? …. well, it is simply expected to be done. Actually, there is no distribution of the work” (JU4)

In summary, for most items the reflections corresponded well with the intention of the scales. However, the items asking about withholding information led to fewer reflections on trust than items asking more directly. Besides, the item dealing with resolution of conflicts included the actual level of conflicts besides issues regarding fairness.

Discussion

The general aim of our study was to assess the content validity of social capital scales included in the widely used COPSOQ II instrument to measure psychosocial work environment. More specifically, we investigated how well the informants understood the questions and were able to answer in a meaningful way, and whether this understanding corresponded with the intentions behind the scales. To achieve this aim we followed a systematic, directed-content analysis focusing on interview quotations on workplace social capital items with particular interest in respondents’ perspective, use of response options, contexts challenging the process of answering, and overall reflections leading to an answer for each item.

Perspective and contextual conditions challenging validity

When establishing group-level constructs based on individual-level data for measurement of cultural phenomena a basic requirement for construct validity is that the questions are formulated in a way which makes it clear for respondents that they are asked about more than their personal perspective and are able to answer in accordance with this wider perspective. Therefore, it is a major finding that for eight out of the eleven items in this study the informants tended to abstract to a group level while answering. This finding corroborates that a shift of referent in these COPSOQ II items can be expected to function in survey situations.

When answering the following questions, however, the informants tended to include primarily their personal perspective: “Can you trust the information that comes from the management?” and “Are all suggestions from employees treated seriously by the management?” For the first of these items the wording differs from the other items by using “you” in a plural sense (Swedish has separate pronouns for singular and plural), which is less clearly underlining the desire of shifting from personal to workplace perspective than when the word “employees” is used as in the other items. A suggestion for future revisions is therefore to use a similar wording as in the other items by using “employees” instead of “you” (in a plural sense). This was changed and tested in further interviews in the Swedish version of the instrument. For the other problematic item we do not have any rational explanation for the finding. However, this fact underlines the relevance of using qualitative methods also for validation purposes.

Social capital can be measured at different levels, from the individual level to that of spatial/non-spatial communities and nations (Harpham, 2008). Even though informants participating in the present study were capable of abstracting from their personal perspective, they frequently took their starting point in their personal conditions or returned to these local conditions in the case of controversy between different organizational levels. This illustrates that although items are formulated to be generic, people interpret the question in relation to what they find meaningful in the context and the knowledge available for answering (Tourangeau and Rasinski, 1988). According to Oksanen and colleagues, little systematic research has been conducted on which level of aggregation would be the most relevant for the measurement of workplace social capital (Oksanen et al., 2013). The results from the interviews suggest that a valid measurement with the current formulation of COPSOQ II items can be established at work unit level.

According to the authors behind the social capital questions in COPSOQ II “workplace” intentionally refers to the company as a whole (National Research Centre for the Working Environment, 2007), but our results indicate that people answer primarily considering their work unit. Today, COPSOQ I & II are used for workplace risk assessments and in research projects (Kristensen, 2010; Moncada et al., 2014). In relation to workplaces a common practice among consultants is to specify what is meant by broad terms like workplace and management by using the actual name of the company and the managers. Obviously, this is not applicable in research projects building on representative samples of respondents. In such situations different solutions exist for a future development of COPSOQ II for improved reliability and validity of a potential measurement at company level: 1) to specify workplace in the introductory text as the company/organization as a whole; 2) to ask analogous questions in relation to each organizational level of interest; or 3) to specify the organizational level of interest as the work unit where the respondent works most of the time.

The main problem of the first option is that it does not follow the logic of informants and thereby might threaten reliability and validity. The advantage of the second option is the opportunities for obtaining more precise information and identifying potential contradictory subcultures at different organizational levels. However, the disadvantage is a lengthier questionnaire and it might also compromise the concept of a generic questionnaire applicable to many different organizational forms. This leaves the third option as the solution, which probably gives most advantages and fewest disadvantages as it is possible to answer in relation to work unit for people working under completely different conditions; it keeps the instrument short; and it supports a low internal non-response as well as a good reliability as it follows the logic of the informants. Besides, by accepting work unit as the level of aggregation, there is still the option for research projects to justify further aggregation to culture for companies (Glisson and James, 2002; LeBreton and Senter, 2007).

Response options

A tendency of an improper endorsement of the middle response option was seen in some cases. This can be understood in the light of the steps of the answering process described by Tourangeau (Tourangeau, 1984; Tourangeau and Rasinski, 1988). When people felt unsure about what was meant by a question [step 1: understand], if the question did not apply to the specific situation of the informant [steps 2–3 retrieve and judge], or if the informant did not find an appropriate response alternative [step 4: report] a clear tendency was either not to respond to the item or alternatively to choose the response option placed in the middle of the scale. Both these reactions impact the reliability and validity of a measurement.

In the scoring of COPSOQ II scales, item non-response is taken into account as at least half of the items of a scale should be answered for calculating a scale score (Pejtersen et al., 2010). In addition, there are other ways of statistically handling non-response, such as symmetric pattern methodology (see e.g. (O’Muircheartaigh and Moustaki, 1999)). A number of techniques for reducing item non-response exist, for example forcing respondents into answering all questions in electronic surveys. Yet such solutions may influence the distribution of responses. While item non-response is a visible problem, an unintended use of response options cannot be either detected or corrected for after data collection with self-administered questionnaires. This underlines the relevance of using qualitative methods in the development of instruments, as problems can be identified and solutions found in time. In relation to COPSOQ a recommendation for practical handling of this problem could be adding instructions to the respondents telling them to skip items they cannot answer or, preferably, offering a response option allowing for having no opinion/don’t know for selected items in future international versions of COPSOQ. In a generic questionnaire it is difficult to ensure that all questions are relevant for all respondents. Therefore, it is in particular relevant to consider an extra response option for items requiring that specific knowledge or experience in work life is required to provide a meaningful answer.

Content of the scales

The operationalization of social capital in COPSOQ II concentrates on trust and justice in relationships at the workplace. In general, the content validity of the scales was supported as informants to a large extent reflected in correspondence with the formulated intention behind the items.

The formulation of items on workplace social capital follows the logic of Schein (1984). From Schein’s organizational theoretical standpoint, behaviour patterns such as withholding information can be understood as the concrete signs of the invisible values and underlying culture (Schein, 1984). Assessing such behaviours is a widely used and recommended method for measuring culture (Glisson and James, 2002). However, the present study revealed that informants seemed to understand items in a literal manner, concentrating on the exact words. For example people paid attention to what specific kind of withheld information they were asked about. This finding points to the importance of identifying the most optimal visible signs of an underlying culture in order to ask straightforward questions central for the phenomenon. A possible alternative way of formulating these questions could be, for example, to ask more directly to achieve openness in communication instead of concentrating on withholding of information. However, considering such changes will need to be addressed in the international COPSOQ society.

Further implications of the findings

The results suggested that the way social capital is measured by COPSOQ II by using a shift of referent can be regarded as valid for measurement of the phenomenon as a group construct rather than an individual construct. In addition, the results suggest that the most reliable and valid measurement can be obtained in relation to work units or local workplaces/departments if using generic terms. These results correspond with findings from health care organizations in Belgium and Denmark, where social capital measured by COPSOQ II has been used at workplace level (Kiss et al., 2014; Lundstrøm et al., 2014). In the Belgian study a social capital measure based on COPSOQ II items had good discriminatory power for small workplaces defined as geographically and organizationally distinct nursing homes, some of which were situated in the same building (Kiss et al., 2014). Corresponding results were found in the Danish study of small medical practices (Lundstrøm et al., 2014) and for schools in the same municipality (Kristensen, 2010; Kristensen et al.,, 2013). These findings corroborating the reliability and validity of COPSOQ II social capital are particularly relevant for future research projects based on multilevel methods, where it is necessary to justify which level of aggregation is the most relevant to collect and aggregate data at and for the choice of theoretical framework.

Risk assessment is widely used to improve work environment, and the importance of relevant instruments of high quality is acknowledged (van Stolk, 2012). Since the 1980s, when New Public Management reforms started in many countries (McLaughlin, Osborne, and Ferlie, 2002), a corresponding trend has evolved to benchmark key indicators with the purpose of organizational development for improved efficiency. A main motivation for companies engaging in such benchmarking activities is a wish to reduce injuries, but also a wish for improved relationships and protection of human capital. However, comparability often constitutes a challenge even though quantitative data appeal to companies (Callen, 2015). This underlines the need for reliable and valid comprehensive instruments such as COPSOQ II in benchmarking activities, in addition to clear procedures. An example here is the well-developed procedure for risk assessment using COPSOQ II in Spain (Llorens et al., 2010; Moncada et al., 2014).

Limitations and strengths

While researchers are experts on theoretical concepts and their operationalization, respondents to workplace surveys are the true experts on their own work environment. The main advantage of using cognitive interviews is that it makes it possible to evaluate how the operationalization of theoretical constructs works in practice and to find solutions for potential problems. Cognitive testing should thus not be regarded as an alternative as much as a complement to other validation methods including e.g. psychometric analyses (Collins, 2015; Willis, 2005). The next step of the validation process of the social capital scales will be based on questionnaire data from workplaces and include psychometric analyses of items and scales in addition to evaluation of fulfilment of criteria for aggregation to group level.

In the current project our aim was to validate the correspondence between the understanding of items among informants and the intentions behind the measurement. However, companies and also researchers pay attention to collaboration and coordination across e.g. departments or occupational groups, which are reflected by concepts such as relational coordination and bridging social capital (Borg, Mateu, and Clausen, 2014; Gittell et al., 2000). These issues are not included in the COPSOQ II measurement, but as the research field of workplace social capital as well as the labour market is changing over time it will be relevant for future international development of the COPSOQ instrument to validate whether the coverage of the domain is sufficient.

It cannot be precluded that social desirability or the interview situation/setting per se to some extent may have influenced the response processes and reflections differently from a situation where the respondent is filling out a self-administered questionnaires. However, we find that it was an advantage that all interviews were conducted face to face. This approach contributed to an atmosphere where the informants described their work life as well as other issues in quite an open way. In the think-aloud paradigm, the role of the interviewer is to interfere as little as possible during the interview, while the role of the interviewer is more active and demanding when probing (Willis, 2005). In the present study we chose a combination of these methods, and the fact that the interviewer had competences in interviewing as well as in work environment theories provided good opportunities for formulating specific probes when asking relevant clarifying questions.

Triangulation by two researchers, who individually summarized the findings after each round of interviews, discussed the findings and presented a summary of problems and solutions to the research group, contributed to a structured and open process concerning decisions on handling of findings. Besides, it resulted in deeper knowledge than would have been possible if only using standardized probes formulated in advance. In addition, it made it possible to develop probing from round to round until saturation in findings was reached.

The purposive selection of informants contributed to ensure that the questionnaire was tested on people coming from a variety of occupations and settings reflecting potential target groups for questionnaire surveys. This was of special importance not only for identifying potential problems but also for the transferability of results, as COPSOQ II is a generic questionnaire applied to different contexts. However, further studies would be desirable in order to confirm the results.

We would have preferred to have the same protocol and procedure for all interviews. In practice, however, the interviews turned out to be rather similar in character as informants in general seemed to fall into two groups: either they spontaneously talked aloud by themselves without much stimulation or probes were needed as openers for stating how they understood an item, where they had problems, etc. None of the informants came up with sensitive personal information as defined by the law; nor did anyone wish to withdraw from the study. Several informants said after their interview that they had found it enriching to think and talk about their work situation in relation to the questionnaire.

The analytic approach of the interviews was twofold. While the initial analyses were primarily inductive and led to a revision of the Swedish wording of the questionnaire, the second round on selected items was deductive and addressed issues of a more overall character in relation to the validity of the measurement. Only minor changes have been implemented in the Swedish version of COPSOQ II in order to secure conceptual congruence and thereby retain the opportunities for international comparisons. Thereby, the more extensive implications of the findings have to be included in the future international development of the questionnaire.

Conclusion

Most often researchers develop scales and items and then test their psychometric properties without discussing the potentially different meanings that respondents can attribute to items or a certain value on a Likert scale. The present study goes a step further than numbers from quantitative evaluations of reliability and validity of COPSOQ II. The qualitative data analyses opened for new insight into how concepts and questions are understood among people coming from different occupations and organizations and situations that might complicate answering.

In general, the findings supported the content and construct validity of the COPSOQ II measurement of workplace social capital as a group construct. First, the content validity of the scales was supported regarding items asking directly about trust and justice, as informants reflected in accordance with the intention behind the scales. However, items on withholding information caused some confusion. Next, the validity of the group construct was largely supported as people in general were able to understand and respond in relation to their workplace rather than taking only their own perspectives into account. Our qualitative analysis of workplace social capital scales included in COPSOQ II corroborates that they are valid measures.