Practical guide to assessing media
30.11.08Practical guide to assessing media
Introduction:
This knowledge resource sets out some of the issues you should work through if you intend to collect information about some aspect of the media. There is a lot of experience to draw from, and what follows below is a guide to some of what exists and how you can use it. It is different to, but complements, other guides such as IFEX’s handbook for campaigning on freedom of expression. It aims to help you get the information you need that can be put to good service with the kind of action tools that the IFEX guide makes available.
Your purposes will determine your assessment tools, and there is a large selection of both purposes and tools to choose from. Your primary aim may be to generate information for an investigative mission, or to brief lawyers and lobby for legal reform. Maybe, instead, it is to assess how officials are fulfilling an Access to Information law, or possibly you want to mobilise for a civil society campaign on how a state-owned broadcaster should act during an election. You might want to monitor progress over several years, and perhaps compare your situation to international benchmarks or other countries’ progress.
Related to your purpose is your scope. If your focus is more narrowly “mass media” rather than the broader “communications”, you may not wish to focus on cellphones (because at this stage of their evolution, they are still mainly used as interpersonal devices). If your focus is “journalism” rather than “mass media”, you are likely to ignore issues like regulating quotas of local content in music programming.
As well as discussing purpose and scope, this report surveys some of the instruments available. In short, it sets out advice around clarifying your focus, and how this impacts upon choosing, creating and using the best tools for assessing some aspect of the media landscape.
1. Surveying the survey tools
No need to reinvent wheels when you decide to do an assessment of media. On the other hand, it does help if you know the range of “wheels” that is on offer. And it helps even more if you know who manufactured them, what size they are, and for what purpose they were originally designed.
Maybe one, or two, models ‘as is’ are directly relevant to your purposes and your context – if so, you’re in luck. But you may prefer to shrink some of them down to the features most relevant to you, or even expand upon features or draw from across several models. In this way, you craft your own system out of this raw material.
Very likely, you will find you need to thoughtfully adapt, rather than simply adopt, so as to suit the particular needs of your situation. For instance, concentration of private media ownership may or may not be highly germane to your context; it may be that state control is the bigger issue in assessing the democratic significance of media. To take another example: it could be that media literacy is your primary concern for emphasis and that you want to elaborate a tool for this. That’s different from say, concerns about media that emanate from specifically taking up the viewpoint of children. From a very wide field, you need to zoom in carefully before you start imitating existing tools.
In looking at what systems already exist, you can also look at their performance in practice. That lets you see if you can milk the findings of people who have already deployed such pre-existing survey instruments. There’s certainly no point in duplicating the data results generated by any of them – unless you think there’s a need to update or that there’s a likelihood of different findings being generated. Otherwise, consider drawing from the information that has already been gathered under these auspices (naturally, giving credit where it is due). It will save you a lot of time and money.
As an indication of what’s happening “out there”, Puddephat (2007:18) has listed 26 different initiatives that define indicators around media. Some are single-issue focused, others are more wide-ranging, and some are even more expansive (e.g. covering the full range of institutions relevant to transparency or corruption – not just the media).
As a sample of some of these, one can cite the well-known Reporters Sans Frontiers’ “Worldwide Press Freedom Index”, and Freedom House’s “Annual Global Survey of Media Independence”. Then there are also systems like the UNDP’s “Guide to Measuring the Impact of Right to information Programmes” (and a homegrown South African version at http://www.opendemocracy.org.za), and Bridges.org’s focus on e-readiness measurement. Another example is the International Standardization and Accreditation Services (ISAS) protocols for “Quality Management Systems: Requirements for Radio, TV Broadcasters and Internet-Content Producers”. There are many, many more.
In deciding if you want to repeat or update a journey, or create a different vehicle altogether to traverse a different map, you may want to consider some of the “wheels” outlined below.
1.1 UNESCO
The recently-devised (2008) UNESCO set of indicators of media development is organised in five categories:
1. Convivial legal environment
2. Plural ownership
3. Democratic performance
4. Capacity issues (skill and supporting organisations)
5. Public access to media
Within each of these categories, there are more specific indicators that have been elaborated. Each also has a “means of verification” and a list of potential data sources. The UNESCO system arose largely from work by former Article 19 director Andrew Puddephat. He reviewed many other instruments, and presented them to a meeting of experts in Paris in 2007. The final result is intended to be a diagnostic tool, revealing national performance across several factors. Importantly, it is not required that every factor should be included in a review. You can be eclectic. That means, for example, that if your concern is media literacy, you can extract these particular indicators out of the overall tool.
The UNESCO schema is highly normative, in the sense of placing value on particular media dispensations (eg. public broadcasting). This means the universality of some aspects of the system does depend on the extent to which you share the same values. On the other hand, because the tool has the blessing of UNESCO, it does carry important political weight that has a bearing on the advocacy-potential of results that are generated by using this schema or at least parts of it.
1.2 Media Sustainability Index
Another instrument that has international resonance is the product of an NGO called IREX (International Research and Exchanges Board). This index has been applied consistently across Europe and Eurasia (starting in 2001, and continued annually since then, covering 21 countries), North Africa and the Middle East (18 countries in 2005), and Africa (37 countries in 2006-7).
IREX states that the index “has evolved into a key benchmark study to assess how media structures change over time and across borders”. It argues that this therefore means the tool constitutes a comparative standard for all countries. However, the Africa study does also expand to embrace what it calls “uniquely African features, including the prevalence of radio—notably community radio”.
The MSI assesses five “objectives” in shaping a successful media system:
1. Free speech and access to public information (legal and social norms).
2. Professional journalism.
3. Plurality of news sources.
4. Independent media are well managed, and allow for editorial independence.
5. Supporting institutions.
Each objective has some seven to nine indicators, “which determine how well a country meets that objective”. The research method combines two features. First, a country is scored by a panel of experts drawn from local media, NGOs, professional associations, international donors and media-development implementers. Second, IREX staff score the countries independently. The two scores are then combined into a final score.
The MSI has a different normative emphasis to that of UNESCO, in that it concentrates especially on conditions for privately-owned media. For instance, it includes as an indicator that “independent media do not receive government subsidies”. Similarly, it upholds sources of newsprint and printing facilities being in private hands, and channels of media distribution (kiosks, transmitters, Internet) also being private.
1.3 African Media Barometer
Formulated by the Friedrich Ebert Stiftung, in partnership with the Media Institute of Southern Africa, the African Media Barometer draws from the standards set out in the Windhoek Declaration (1991) and the African Commission on Human and Peoples’ Rights Declaration on Principles of Freedom of Expression (2002). This tool covers four sectors:
1. Freedom of expression.
2. Media landscape characterised by diversity, independence and sustainability.
3. Broadcasting: transparent and independent regulation and true public broadcasting.
4. High levels of professional standards.
Each of the four areas has approximately seven indicator areas. The research method is via national panels of up to 10 people, half of whom are directly involved in the media, and the others from civil society. The Barometer operates with a scale in terms of which each country is scored in terms of the extent to which it meets aspects of a given indicator. The scores for each indicator are given equal weight when added up and averaged.
Given the reference points of this tool in credible African declarations, it has the advantage of measuring performance against self-proclaimed continental standards. Its normative character is evident in the importance it attributes to broadcasting in African countries (It makes the topic a sector area in its own right and of equal weight to the other three somewhat broader categories.) The Barometer also includes the requirement that “the state broadcaster is transformed into a truly public broadcaster”.
1.4 AMDI, STREAM and AMI
The origin of these tools was with the Commission for Africa initiated by the former British Prime Minister, Tony Blair. The “African Media Development Initiative” (AMDI) was overseen by the BBC World Service Trust in partnership with two African universities. This project researched 17 African countries, answering three questions:
1. What are the key changes/developments in the media sector over the past five years?
2. How have media development activities contributed to the development of the media during this time?
3. What contributions are likely to make the greatest impact on the development of the media sector within sub-Saharan Africa in the future?
To answer these, AMDI looked at:
1. Extent of the “media sector” (which it interpreted not only as covering television, radio, newspapers, but also as the Internet and mobile telephony).
2. Media support: production agencies, market research companies, training, and NGO activity.
3. Media legislation and regulatory reform.
4. Technology and equipment.
5. Professionalisation.
6. Local content production.
Research was done through accessing pre-existing data and through qualitative interviews with key stakeholders and experts in each country.
The “Strengthening African Media” (STREAM) was carried out parallel to AMDI, and overseen by the UN Economic Commission for Africa. The research method here was consultative conferences in the five different regions of the continent, drawing participants from 30 countries. It covered:
1. Media practice
2. Policy and legislation
3. Media ownership
4. Media training
5. Media support
Fusing the two approaches, the “African Media Initiative” (AMI) came up with three very broad ways to categorise the key features of the media landscape. In effect, they were: (a) Contextual politics (especially media freedom and policy), (b) Economics of media markets (including investment climate and technology), and (c) Capacity (professionalization, support groups, training).
These three initiatives (AMDI, Stream and AMI) were driven by concern to make a case for support for media development activities across Africa. This is why they attempt to provide a holistic focus – covering every essential base to ensure that progress in one area is not blocked by obstacles in another. In this attempt to be comprehensive, the three approaches differ from single-issue focus tools, such as the Committee to Protect Journalists which monitors statistics on journalists killed. They are also much broader than, say, Freedom House (http://www.freedomhouse.org) which looks at three wide areas: legal environment, political environment and economic environment – but from the particular and exclusive vantage point of what is relevant to media freedom.
1.5 Media Legislation in Africa. A Comparative Legal Survey (2007)
This study was undertaken for UNESCO, and it evolved a template which flowed from international instruments and principles (including specifically African ones) inasmuch as these pertain to media law reform. A total of 32 points of focus were grouped under the following categories:
1. Relevant constitutional and contextual provisions.
2. Laws relating to the status of journalists (do they need to be registered or licensed?).
3. Laws and regulations on licensing media (print, broadcast).
4. Laws on ownership legislation (eg. limits on cross- or foreign ownership).
5. Other media-relevant laws covering state secrecy, subsidy, defamation, etc.
6. Laws on reporting courts.
7. Laws and regulations on media and elections.
8. Ethics and the law (statutory and non-statutory, including provisions on the right to reply and confidentiality of sources).
9. Actual respect for freedom of expression and law by governments, media and other actors.
1.6 UNESCO Criteria for excellence in African journalism education
Also specific to African conditions, this tool was developed out of an online consultation with 19 African journalism schools. It also, however, drew from accreditation systems operated in the USA, UK and Francophone countries. The result pinpoints three broad criteria areas for assessing excellence in the African schools:
1. Internal: Curriculum with theory and practice, and specified learning outcomes.
2. External: Professional and public service, and external linkages and responsiveness.
3. Future: Existence of a mid- or long-term strategy of development.
Within these areas, the system includes indicators particularly relevant to Africa – such as the significance of the work done by a school in regard to key issues on the continent such as democratisation, HIV, Pan Africanism, and multi-lingualism.
Application of this system (during 2007) was through a combination of schools doing self-scoring and site visits by independent researchers. The result was that 12 of some 200 African journalism schools were identified as “potential centres of excellence”, and UNESCO was then able to target its support on helping to realise this potential.
1.7 AfriMAP Survey on Public Broadcasting (2008)
Supported by the Open Society Institute, this current survey builds upon the 2005 “Television Across Europe” study of broadcasting regulation, policy and independence in 20 European countries (see http://www.mediapolicy.org). This latest study, under the Institute’s AfriMAP initiative, surveys 12 African countries, and it covers “both public broadcasting as an institution and public interest programming which can be offered by all services across the broadcasting spectrum”.
Among the indicators being investigated with this tool are:
• The impact of digitalisation on broadcasting.
• Funding of the state/public broadcaster.
• Programming of the state/public broadcaster.
• Perceptions of and attitudes/expectations towards the state/public broadcaster.
• Public interest programming offered by non-state/public broadcasters.
• Comparison: Public interest programming offered by state/public broadcasters and other services.
2. Criteria for choosing
Puddephat (2007:10) points out: “For example, five Middle Eastern and North African countries categorised as “near sustainability” by the Media Sustainability Index (MSI) are classified as “not free” by Freedom House; the Palestinian Territories are, according to MSI, “near sustainability” yet come second to bottom of the Freedom House scale, just above Libya.” This observation highlights that it is not a simple or neutral matter to choose one particular tool. There are consequences. Puddephat himself has compiled a valuable comparison of 15 instruments in regard to what they cover (2007:42-3). His tabular representation enables you to see at a glance where there is overlap (for example, eight of them take on board defamation laws), and where there are gaps (only three look at access to printing and distribution facilities). Adopting a multiple approach – or at least doing a cross-tool scan before alighting on one tool – is one way to deal with the fact that different tools on their own can produce rather different results.
Another issue relevant to choosing from existing instruments is to be aware that many date from a pre-convergence view of media. This context means they may have some blindspots:
• They do not always transcend the idea of separate silos of media, and they may therefore miss out on important developments at the level of production, ownership, distribution, regulation and consumption.
• Many are also narrow in the sense of ignoring cultural industries (like music or film), and intellectual property issues more broadly. For tools to study the latter, for example, you may need to go outside the “predictable” sources such as those cited in the section above, and look instead at resources like the “WIPO Guide on Surveying the Economic Contribution of the Copyright-Based Industries”.
• Many of the media instruments discussed above also date back to a period when “media” equated to (specialised) institutions, whereas today the actors involved in mass communication increasingly include numerous other players. You may not want to ignore the significance of these other mass communicators no matter whether they are individuals, NGOs, public institutions and companies. They could be playing in the public sphere with great relevance to public opinion, alongside the traditional media. The point is that media is no longer the exclusive preserve of “the media”, although it is of course still important to acknowledge the latter as institutions where a distinctive kind of mass communication is core business (as distinct from being incidental or secondary).
• In addition, whereas the national unit of analysis still has much relevance, many communications issues today are best understood in terms of transnational, international and global connections that encompass technology, economics, policy regimes and content flows.
• Questions also arise today as to what constitutes “journalism” within the burgeoning “mass communication mix”, given the passing of a period in which the practice was coterminous with fulltime “professional” journalists. There are also issues around the “former audience” which in many cases is not a passive and atomised mass of individuals, but a meaning-making and –shaping set of shifting communities and social networks.
All these fast-changing dynamics complicate the development and utilisation of indicators, meaning that caution is needed in adoption of one or more existing systems. Drawing from Puddephat (2007:20), this is where the value of a “toolkit approach” comes in. The advantage of such an approach, in his view, is that it “offers an inclusive list of indicators and methods from which selections can be made according to the requirements of a particular programme or intervention”. In addition, it “recognises that indicators and methodologies must be customised, using local expertise to fit the particularities of the national context”. He adds that “indicators must be tailored to the correct level of engagement within each national context (e.g. the national media system, the individual media organisation, the professional group)”.
The information that follows below offers some guidance as to how such adaptation can be done.
2.1 Definition and scoping: where do you want to go?
A starting point for choosing amongst existing “wheels”, or in wanting to design one’s own, is to be very clear about what part of the landscape you wish to map. Many of the tools described in the section above fall short of properly scoping and defining their terrain. But there is a big difference between whether you are looking at “communication development” broadly (which might include community internet centres), or more narrowly the development of the media industry (and which may include the development of a market research and advertising industry). There is also a big distinction between looking at the latter area (“media development”), and looking at “media for development” and “development media”. The first focuses on developing the media as an end in-itself; the others relate to the role of (some) media as a means to an external end (eg. Citizenship development, corruption reduction, HIV-Aids awareness and behaviour change, etc.).
In this context, it is particularly important for you to decide what constitutes “media”, and why you want to look at it. Is it all aspects of content generation, or is it mainly the role of journalism or, say, of edutainment – and why? Does it include folk media (present in popular culture)? Should it encompass cellphones (bearing in mind their current “limited” mass media character)? Start with the widest definition of media, and then decide logically what and why you want to highlight within this.
2. Putting values in the driving seat.
A second point in choosing or redesigning a survey tool, is to be clear and upfront about the normative premises that you stand for. For example, perhaps your focus is on private radio station entrepreneurship, rather than, say, community participation in community radio. Different priorities reflect different beliefs about media and society.
All choices reflect values about what you regard as significant – so recognise this, state your orientation upfront, and give the reasons for that focus.
One technique to consider is distinguishing “media density” from “media development”. For instance, the first phrase can be relatively technically defined (eg. a common currency of the number of occupational journalists per 1000 population). Then normative interpretations of the second phrase (“development”) can then follow from this – for instance, how independent and professional these journalists are.
In addition, one can use the less-loaded phrase “media assistance”, rather than “media development”, when talking about interventions in regard to changing the quantity and/or quality of the given “info-structure” and its character.
Being clear about the normative basis of your project, and the normative assumptions embedded in existing tools, not only puts you in the driving seat, it also clarifies the starting point of your journey and what kinds of aids are appropriate to your purpose.
2.3 Surfacing assumptions about cause and effect.
A third element in toolkit design is to make a conceptual distinction between means and ends. The means is looking at the state of something in terms of its significance for something else. UNESCO for its part describes its five areas as “outcomes” of media development, and yet some of them (eg. Conducive legal environment) are more properly means towards an outcome (eg. Democratic performance of media). It all depends on whether you interpret “media development” as meaning interventions into the environment of media, or directly on the media as such. The UNESCO scheme puts these all together. But it can help a lot if you are clear on the different statuses of your means and ends, even though they are interrelated.
To give an example on this issue, donor support for community media is often a means towards constructing a sector that in turn plays a particular role in society (eg. local democracy and development). The end goal is that role dispensation. But you can break this down into at least three distinctive areas which merit investigation and intervention:
• One study might focus on the resourcing available to community radio per se (as a necessary, though insufficient, precondition for the penultimate goal to be achieved).
• Another initiative might address the shortfall between the ideal democratic roles of community radio, and its present capacity and orientation to do so (the ultimate goal). This might, for example, look at the extent of participation.
• A third approach could concentrate on the issue of an enabling legislative environment for community radio – which in most respects is not an absolute end-in-itself (although it could still be an essential goal to achieve). First and foremost, therefore, the environmental issue is mainly a means to reaching the conceived ideal role of community radio in society.
All these three foci are absolutely legitimate. The point is simply to highlight the value of distinguishing the items in a wider chain of assumed cause and effect. You do not want an undifferentiated set of indicators, where the status of each in relation to the other is uneven or unclear. The inter-relationships (even if assumed) between means and ends, and how those ends then serve as further means to bring about yet further ends, need to be unpacked. In that way, you can prioritise your precise point of focus in the totality of the chain.
4. Utilitarian considerations
Your mapping project should be driven by what it defines as its very own ultimate goals in contributing to understanding and intervening in the chain of impacts. In this regard, you want your enterprise to have impact, and for this you need to think credibility issues in choosing instruments. This is why drawing from accepted international instruments like UNESCO’s can be very powerful. Utilising international standards is also a way to enable international comparisons and benchmarking.
Also, don’t forget the value of taking a medium-term time frame in assessing how to maximise the impact of your enterprise. Will the instrument/s you use be valid over a period of time, allowing you or others to monitor progression or regression in the media?
Affecting your conceptualisation here will also be the extent to which you are looking at snapshot information, or trend information. For instance, rather than mapping HIV-Aids messaging in public broadcasting after a particular campaign, you want to examine the more ongoing themes over a year’s output. The point is to keep your practical purpose as a guiding star in selecting or designing your system.
5. Discreet indicators
As Puddephat (2007) advises, it is important to ensure that your indicators are separated out to address one key issue at a time, and do not blur several elements in one category or conflate distinctions between different units of analysis. He also advises that indicators be structured to be gender-sensitive and pro-poor, so as to enable you or others to disaggregate findings along these axes.
6. Pragmatics and logistics
Another issue that should affect your selection and design of indicators is the practicality of them. Puddephat (2007) suggests that quantitative measurements whenever possible are preferable, but he also qualifies this by saying that this requires that measurement data be sufficiently reliable. He points out that for many indicators in many countries, data “doesn’t exist, is inaccessible, is out of date, is inconsistent, or a combination of all of these”. This situation sometimes explains why people resort to alternative methodologies based on panels of media professionals using qualitative assessments. Such findings can also be partly used in a form of “triangulation” (see section 4 below) with quantitative data. One limitation of the panel or focus group approach is that the judgements are inevitably based on perceptions at a particular historical conjucture. This means that the findings are not easily generalisable. Nevertheless, they can assist in extrapolating broader principles even if not be mistaken as being tools for making other kinds of claims.
7. Think long-term
Sustainability (in the sense of repeating the exercise) is important, and that in turn relates to available budget now and in the future. Once you get to your destination, the question is: where to next? Will you want to revisit it again in a year’s time? It helps if you can answer this question before you even begin, and to develop contingency plans accordingly.
3. Implementing
In conducting an assessment of media development, it is valuable to distinguish between quantitative and qualitative research. They are distinct, although at the same time there is necessarily a qualitative dimension to defining what is being counted in a quantitative exercise. Likewise, some qualitative research can involve counting and scoring. The difference lies mainly in their different knowledge claims. Quantitative findings are usually supposed to be representative of a broad reality, and quantitative samples are legitimately generalisable to a wider universe. Qualitative research goes deep, rather than wide, using case study approaches for example, from which abstract principles can be extrapolated but not statistical trends.
It is helpful to “triangulate” (see below) quantitative and qualitative data in order to arrive at a fuller picture of the media reality you are focusing upon. As such, there is need to distinguish between the two assessment methodological designs and establish their relative strengths.
Qualitative research:
Much of the data produced by the media development assessment tools cited above is qualitative. There are instances in which assessors seek to “quantify” the responses. This can only be for ease of data presentation and interpretation, however. It allows for drawing comparisons, but does not indicate any statistical representation, nor does it show interval or ratio levels of measurement. In quantitative research, an interval level of measurement means that numbers assigned to research categories and/or indicators are meaningful as numbers and not as labels; ratio levels of measurement include an absolute zero point, as in when we measure numbers of radio and television stations and sets per country, etc.
In the absence of statistical information, the qualitative assessment of media development indicators tends to be based on nominal and ordinal levels of measurement, whereby numerals are only assigned to categories and indicators for the purpose of classification and rank-ordering. As a result, such scores do not represent the actual state of affairs. For example, the numerical measures of “media diversity” in the African Media Barometer do not represent the actual state of affairs on the ground. They represent a group’s perceptions. But the scores assist us to classify, rank and compare such perceptions over time, and in relation to other groups’ scores on this point in other countries. The findings for one national group ought never to be assumed to apply to other countries (even with similar objective features), and neither as being necessarily representative of how the majority of people in the particular country perceive the issues.
The trustworthiness of qualitative assessments, then, lies within the concepts of credibility, dependability and conformability. These can be achieved by the following:
• Careful use, interpretation and examination of appropriate literature.
• Careful justification of the qualitative research methodologies employed.
• Careful structuring of the data analysis to ensure full and descriptive evaluation and assessment, particularly in relation to data of key significance (in Levy, 2006:383).
Qualitative data, for the most part, are good at helping us to assess the “thickness” of the description and interpretation of media reality. Such findings can assist us to record and analyse the different perceptions and/or discursive practices associated with the respondents and examine what might colour their responses. In this way, qualitative assessments help us to develop keener understanding of the media landscape. So, where our research interest lies in developing a nuanced understanding of media reality, qualitative research of the type used in most of these assessment tools is certainly useful.
Quantitative research:
In most cases, quantitative data are credited with more believability than qualitative data, because of their presumed character to represent a state of affairs in an incontestably measurable way. Quantitative research can answer the more positivistic assessment questions about “reliability”, “validity” and “bias”. In media research, this is often associated with establishing, for example, the quantity of media plurality as opposed to its quality. While quantitative research can easily adduce statistical evidence about how many media outlets a country has, it is not always easy to find generalisable samples that deal with more complex matters such as media performance, or to reduce something like media impact to common features that can be counted.
For this reason, qualitative research can often better assess, for example, a different dimension to media plurality – such as whether the number of media outlets translates into participation in media production, associated expansion in freedom of expression, gender empowerment, poverty alleviation, and the like. However, it is often important to establish the quantitative nature of media reality before we can qualitatively assess its professional and other aspects.
As noted above, it can be a challenge to gather quantitative data in most of the developing countries, where bureaus of statistics are not sufficiently developed. Another problem has to do with little or no media research capacity. But mention must be made of the Pan-African Media Research Organisation (PAMRO) as well as the Steadman Research Group. At least in Africa, these can be used as a first port of call for media development research of a quantitative statistical nature.
4. Quality assurance
For purposes of quality assurance, there is a need for media assessments to be subjected to “external” critique by experts or people who are “disinterested” in the assessment project. Such experts or people might be located in academic or research institutions. They should be drawn in right at the stage of research scoping and instrument design, and not after the event. If your assessment is to withstand criticism, you need advice about anticipating what kinds of knowledge claims and methodological limitations are appropriate to it.
In more specific terms, quality assurance can be obtained via the following processes:
Peer review
Here, the assessment tools, the quality of the data and the results of the assessment need to be subjected to some “dispassionate” critique or review. This should be treated as part of the normal research validation process. Indeed, the reviewer reflections can become an integral part of the assessment results.
Triangulation
The example of combining qualitative and quantitative research given above (in regard to media pluralism) exemplifies what “triangulation” is all about. According to one writer:
Because of the complicity in “a confused reality”, it is difficult to study/investigate a phenomenon in its totality. In this complex reality, multiple methods … afford a partial solution… (Grobbelaar, 2000: 91-92).
In the type of qualitative assessments undertaken by the various media development tools cited above, it is clear that triangulation with quantitative information could make for richer results. At the same time, one should not think that quantitative and qualitative findings can directly corroborate each other: they are only complimentary animals.
Where triangulation can occur a more corroborative way is within quantitative, and within qualitative, research:
• Within quantitative research, triangulation would serve to further assist in reliability and validity. Here, reliability means that if identical investigations are repeated, similar research results will be obtained. Validity means that the assessment tools as well as assessment results and interpretation are in sync with the research aims and objectives. In quantitative assessment, you can try to control for some intervening variables in a way that can ensure a greater degree of reliability and validity.
• On the other hand, qualitative research, as noted above, is more interested in the thickness of description and interpretation. It is aimed at creating greater understanding of the phenomenon under investigation. Triangulation with other qualitative research thus becomes important, at the level of extrapolation and principles uncovered.
To assess “media development”, it might also be useful to contemplate triangulation at different levels, as suggested below:
Methodological triangulation: Here, both quantitative and qualitative research designs should be contemplated, in keeping with the overall research aim and objectives. An attempt at this was evident in the AMDI and STREAM research processes – a combination of literature review, personal interviews, case studies, consultative workshops, and the like.
Ethnographical triangulation: Here, the assessment effort must be targeted at different people in different discursive practices. More often, assessments of this type, especially those undertaken in Africa, have tended to focus on the same respondents. This has the danger of the “Hawthorne effect”, with respondents knowing exactly how they ought to respond during focus group discussions. Why, for example, do certain categories of respondents repeatedly describe the media as tools for political repression? What is their discursive practice in society? What happens when such respondents get into positions of political power? How can such respondents’ responses be checked against other readings?
This “Hawthorne effect” tends to skew the results in favour of the “anti-establishment” discourse. It is thus important to cast the net even wider and draw in as many voices as possible as an attempt at “balancing” the assessment outcomes.
Geographical triangulation: As a result of the often logistically determined focus on the same respondents, it might be important for media development assessments to go beyond the line of rail and focus on other geographical localities, especially rural areas. Although this might be problematic in terms of logistics and costs, it is something worth investing in, so as to draw in “ordinary” and poor people’s voices and perspectives. This is particularly appropriate in an environment calling out for with rural-based community media initiatives.
Gender triangulation: It is usually men who speak whenever there are assessments of the type referred to above. There should be a deliberate effort to draw in women so that they can speak on issues that directly relate to them. This is a process of empowerment, in itself, which is a critical aspect of measuring media development. To what extent are women represented in media content? Who speaks for them? How are they represented? These are legitimate questions which need to be factored into the design of any assessment tool.
5. Interpretation and report writing
Given the need for thick description and interpretation of mostly qualitative data produced using the assessment tools cited above, it is important to undertake a series of checks and balances in the interpretation of data and report-writing. These are:
• Checks by members of the group, i.e. asking respondents to comment on drafts, facts and their interpretations of those facts.
• Initiating independent audits of the assessment tools, data and interpretation. This can be outsourced to academic and research institutions.
• Having a number of interviewers carry out interviews, followed by interviewers discussing the meaning and interpretations of the data.
• Presenting the findings of the research study to the original respondents in a focus group meeting and then inviting respondents to comment and provide feedback and discussion in relation to the findings (Levy, 2006).
Here, the emphasis is on ensuring a greater degree of credibility, dependability and conformability (Levy, 2006).
6. Publicity and advocacy post-publication
It is clear that one of the purposes of most of the media development assessments is to influence media and communication policy, especially in developing countries. It is also meant to influence the way media is generally produced. This means that such assessments are in fact action research, with their main emphasis on the following:
• The people or organisations that are studied actually take part in the research process itself.
• Inclusion of ordinary and everyday knowledge.
• Empowerment of the subjects of research.
• Awareness creation.
• Linked to political realities or policy-making processes (Grobbelaar, 2000: 85; cf. Babbie & Mouton, 2001).
The published results of research need to be widely distributed to different stakeholders that might find them useful. One could go further and organise launch workshops at which the findings can also be discussed and interrogated. This, in and of itself, becomes another process of further validating the research findings with interested parties. More often, however, such assessments end up without being actively taken up further and serving as effective tools for policy and legislative reform.
An interesting idea might be to organise a dissemination workshop with parliamentarians, especially in developing countries, who might the research reports useful for their legislative function. The possibilities are limitless but it is important to plan for them well ahead.
In sum, an assessment of your mapping exercise should start long before the actual research is conducted, and its lifespan should persist long after the findings are written up.
Conclusion
This report firstly lists several “models” of media development assessment tools. These include:
• The UNESCO indicators of media development
• The African Media Barometer
• The AMDI, STREAM and AMI processes
• The comparative legal survey of African media legislation
• The UNESCO Criteria for excellence in African journalism education
• The AfriMAP Survey on Public Broadcasting.
Each of these tools has specified criteria and indicators for “measuring” media development, and within their chosen scope and normative frame, most aspire to be as comprehensive and cohesive as possible. In many cases, the focus has tended to be on assessing particular dimensions across the totality of the cultural circuitry of media production, representation, consumption and regulation. Aspects may be relevant to your own particular purpose.
However, the report recommends the need for a clear decision about the scope of the assessment; a consciousness of normative value positions; clarity about the cause-effect aspects of the assessment; clarification of utilitarian concerns; development of discrete indicators of measurement; practicality of the assessment tool; and longevity of time-frame.
Secondly, the report addresses the question of implementing an assessment exercise. Here, it tackles the challenges and benefits of triangulating. This is relevant both within and between qualitative and quantitative assessment approaches. The argument is made that each of these methodological approaches is important to the task of assessing the complexity of media development. Although most media development assessments tend towards the qualitative approach, this study calls for a purposive triangulation that incorporates aspects of methodological, ethnographical, geographical and gender triangulation. This is aimed at enhancing the assessment tool’s representativeness, credibility, dependability and conformability.
Thirdly, as a way of further enhancing data quality assurance, the report deals with the value of a “ethnographic” processes of developing the systems, and also in interpreting the results of the assessment. This helps ensure that the final assessment product is “owned” by the various stakeholders involved in the assessment process. Associated with this collectivist process of data interpretation and report-writing is the need for developing and elaborating clear post-publication publicity and advocacy plans and activities. Such post-publication planning should not be seen as purely instrumentalist; it is also an organic part of validating the findings of the assessment exercise.
It is clear that assessing media development is much more than just a research activity; it is a conceptually-informed process of enquiry and a conscious act of intervention in remedying the problems brought to light as a result of the assessment exercise.
List of references
Babbie, E & Mouton, J. 2001. The Practice of Social Research. Cape Town: Oxford University Press.
Berger, G. 2007. Media Legislation in Africa. A Comparative Legal Survey. Grahamstown and Paris: Rhodes University and UNESCO.
Grobbelaar, T. 2000. Types of Research, in Research in the Social Sciences. Pretoria: University of South Africa.
Levy, D. 2006. Qualitative Methodology and Grounded Theory in Property Research. Pacific Rim Property Research Journal 12(4): 369-388.
Puddephat, A. 2007. Defining Indicators of Media Development. Background Paper. Paris: UNESCO International Programme for the Development of Communication.