MISO Product Testing in a Non-Permissive COIN Environment

MISO Product Testing in a Non-Permissive COIN Environment

Brian J. Hancock

Introduction

Modern counter-insurgency (COIN) strategies are a complex cocktail of lethal, non-lethal, and stability and support operations (SOSO).  In order to achieve sustainable success each element must be precisely balanced with every other.  When employed correctly Military Information Support Operations (MISO--formerly PSYOP) as a force multiplier enjoys unparalleled capability to shape, and ultimately win, the hearts and minds of the populace who are the key terrain in any COIN conflict.  By the same token, MISO improperly employed, can have a serious and deleterious effect on the overall war effort.  It is for this reason that release authority for Military Information Support Operations comes directly from the Secretary of Defense.

Successful MISO Commanders must answer a series of Priority Information Requests (PIRs) which include: Is MISO product effective?  Could it be more effective?  Is it reaching the target audience (TA)?  Is the population ready to receive and act on the next product in the MISO series?  Is there anything offensive in the product which might turn the population against the coalition?  Can the MISO product be subverted by the enemy?  The answers to all of these questions can be obtained through a deliberate and aggressive program of MISO product testing.

MISO product testing is an essential, and often overlooked, part of the Seven Step MISO Process.  By doctrine, MISO Product Pre-testing is to be accomplished in Phase IV, Product Development and Design.  Post-testing on the other hand is relegated to Phase VI Production, Distribution, and Dissemination.  Due to severe resource constraints, the all too frequent reality in the combat zone is that focus is placed on product creation and distribution, rather than on planning and evaluation.  As administrative processes are cut product testing, which many consider superfluous, is often one of the first to go.

This trend is further exacerbated in austere third world countries such as Afghanistan, and similar areas in Africa and parts of South East Asia.  Much of the MISO doctrine presented in FM 3-05.301 Psychological Operations Tactics, Techniques and Procedures was developed for a more traditional conflict in ideal conditions characterized by excellent infrastructure, freedom of movement, high literacy rates, and easy access to the target audience (TA).  By contrast, much of Afghanistan is composed of inhospitable human and physical terrain.  It is not uncommon for a population center that is only ten miles distant to require in excess of a day to travel to—much of it through enemy controlled territory.  As a result, mission opportunities are reduced and few actually involve interaction with the indigenous population that forms the target audience.

Despite these challenges the returns in MISO product effectiveness and risk mitigation more than compensate for the additional resources required to conduct successful testing in non-permissive environments.  This article will define the essential components of a successful MISO product testing program as well as provide best practices to ensure mission success.

Pre-Testing

Summary

Investigations into MISO product which fails to achieve the desired effect, or worse still backfires, often attribute the lack of pre-testing as a primary cause.  Properly conducted pre-testing not only allows the Tactical Product Development Detachment (TPDD) to determine if the MISO message is understood and accepted by the TA, but also detects elements which may be culturally offensive, as well as elicits feedback on how the product can be made more effective.

The current version of FM 3-05.301 outlines the basic processes and methods for conducting successful product pre-testing.  The recommendations are somewhat idealized, as few Tactical MISO Companies have the resources required to pursue the full testing process as presented.  Probabilistic sampling strategies which require knowledge of techniques such as the Kirsh Grid Method, software such as SPSS, and a background in fundamental statistics are beyond the capabilities of most MISO elements.  MISO in theatre often finds itself under tremendous time pressure, with too few personnel resources available, to conduct pre-testing.  This requires abbreviated methods which are not adequately addressed in the FM.

Sampling

Proper survey construction requires careful attention to how sample populations are generated.  A sample population is a subset of the overall population that provides an indication of the attitudes and behaviors of the population as a whole.  There are a variety of methods for generating valid sample populations.  Such techniques fall into two broad classifications, Probability Samples and Nonprobability Samples.

Probability samples are carefully crafted in advance and designed to capture a true representative slice of the larger population.  These methods are time consuming and require extensive preparation and resources.  While they tend to produce results with greater scientific validity, they are rarely suitable in the COIN MISO environment where short suspenses and a high degree of resource constraints are the norm.  The three most common Probability samples are the Simple Random Sample, the Stratified Random Sample, and the Cluster Survey Sample.

The Simple Random Sample is characterized by an equal chance of each person in the Area of Operation (AO) being included in the sample.  Sample names are generating from voting lists, or census data.  Unfortunately in third world countries like Afghanistan, such lists are rarely available, making this technique difficult to employ.

The Stratified Random Sample is similar to the Simple Random Sample but involves more groups.  To employ this technique at a University for example, each grade level might be broken into separate strata, each of which would then be subjected to a Simple Random Sample.  Once again, the limited availability of data in third world countries, makes the Stratified Random Sample labor intensive and costly to employ.

Cluster Survey Samples break the sample population up into progressively smaller blocks.  For instance, at the highest level the Regional Command data or population might be sampled.  Upon completion of this sample, a smaller area or Province might be sampled, followed by each district that composes the Province in turn.  Each sample would then be polled and the results compared.  This technique not only requires robust census data, it also requires a great deal of time to poll each sample.

Non-probability samples are less resource intensive, and better suited to the needs of Tactical MISO.  Unfortunately, these samples are less accurate than those generated by probabilistic techniques.  The two basic non-probability categories are the Accidental Sample, and the Quota Sample. 

The Accidental Sample is the easiest method to employ, and consequently is used more than any other by MISO. To conduct an accidental sample all one has to do is stake out a particular location and sample every person who passes through that location.  Locations have to be chosen very carefully to survey the broadest mix of the local population possible in the period of time allotted.  If the sample location is in a very rich, or a very poor neighborhood, it will over represent a particular segment of the society and skew the results.

The Quota Sample, while employed somewhat less frequently than the Accidental Sample, is still practical for certain MISO Operations.  In this technique a quota for each subgroup of a society is set, and once that number has been interviewed no more of that subtype are surveyed.  A simple quota sample for instance might be to interview 100 rich people, 100 middle class people, and 100 poor people.  To obtain these results the testing element would likely have to visit multiple locations in order to find areas of high concentration of the quota group sought.

Survey Design

The two most critical data points that must be ascertained in a MISO product pre-test survey are; does the TA understand the message in the product, does the TA accept the MISO argument presented.

It is also advisable to query the TA to determine if they find elements (art, color, images) in the product offensive and ask if they have any suggestions to make the product more effective.  Sample questions to ascertain these items for both print and radio products are presented below.

Print Products

• What is the meaning of this product?
• What does this image urge you to do?
• Is there anything offensive in this product?
• What do you recommend to improve this product?

Radio Messages

• What is the meaning of this message?
• What does this message urge you to do?
• Is there anything offensive in this message?
• Do you have any suggestions to improve this message?

It is also important to remember that how a prospective person is interviewed can have a dramatic effect on the answers received.  All subjects should be treated with respect and efforts to build initial rapport should be utilized.  Subjects who are busy and preoccupied are quite likely to tell the interviewer whatever they think the interviewer wants to hear just to get rid of them.  If a subject’s diction, tone, or body language give the impression that he is not being sincere, the interviewer should note this on the survey.

Methodology

Initial pre-testing guidance for each MISO product is contained in its corresponding Product Action Worksheet (PAW).  As PAWS are typically produced by under staffed Tactical MISO Detachments (TMDs) which are falling in on Areas of Operation twice the size of what they are designed to support, the quality of the paperwork they produce is often low.  It is not unusual for test questions to be omitted, or to contain inappropriate queries for proper product testing.  The Company (CO) level MISO Element should correct these deficiencies in the PAW and continue the testing process.  As a point of courtesy and training, the corrected PAW should then be sent back down to the TPD, in order to (IOT) provide guidance.

The first consideration when testing product is to have a firm understanding of the time suspense and the resources available.  At the most basic level pre-testing strategies can be broken down into Standard, Abbreviated, and Emergency classifications-- each of which contains one or more phases.  Testing itself is further subdivided into Phase I, Phase II, and Phase III.  The 307^th Tactical PSYOP CO helped pioneer the techniques espoused in this paper.  They deployed to Kandahar Afghanistan and utilized an Excel based tracker to monitor the progress of product testing which was stored on the Division share drive for easy access.  As products moved through each phase of testing, the testing tracker was updated.  This allowed the Testing Group to show current status of all items in the testing queue at any given time. 

Phase I testing consists of all the MISO Interpreters.  They should be tested individually and not be permitted to come to a “group” opinion.  As at least one of the translators was involved in creating the product, their point of view is clearly biased.  This is one of the reasons why it is critical to interview each translator privately.

Phase II testing utilizes a panel of experts who are generally polled as a group.  The group is typically composed of cultural advisors, the Department of State, a battlespace commander, people with experience in MISO and public relations, and other subject matter experts.  This type of product review is frequently referred to as a “murder board”.  MISO has an ongoing need to conduct product testing, and assembling a panel of experts quickly, is rarely possible.  Phase II testing is most effective if a weekly meeting is established to facilitate the regular participation of the experts. All accumulated product undergoes testing at each meeting.

Phase III testing is the most exhaustive, the most important, and requires the potential MISO product to be tested with the intended Target Audience.  It is critical to understand where the specific TA congregates so that sample testing locations can be identified.  For instance, if the TA for a given product is the “Afghan People” then testing the product at an Entry Control Point (ECP) frequented by Pakistani truck drivers will return invalid results.  By the same token testing products which specifically target Anti-Afghan Forces (AAF), with the general Afghan population, is unacceptable.  While there is some value in ensuring that a member of the general population, who inadvertently gets a hold of a MISO product targeting AAF will not be offended, they will not understand the context of the MISO message. 

Site selection for MISO pre-testing is an art and a science.  In general:

Places that Do Not Pre-Test Well

• DFACS/ Public Common Areas on a FOB
• Bazaars on a FOB
• Entry Control Points Used by Foreigners
• Mosques

Many dining facilities hire local nationals (LNs), and public common areas are typically maintained by a LN custodial staff.  As these individuals are direct employees of the coalition, any information they provide is biased.  In addition, most will get into trouble if they are seen talking while they are supposed to be working, so they are likely to respond with whatever will get rid of the interviewer the fastest.  The only LNs present in a Forward Operating Base (FOB) bazaar are the merchants, who are too busy hocking wares to engage in product testing.  They are also shrewd and will consistently tell the interviewer whatever they think will make them happy in order to put them into a good mood and thereby increase the chance of making a sale.

Places that Pre-Test Well

• Base Support Companies
• Local Cafes (Kabob House etc)
• Local Airports
• Entry Control Points Used by LNs
• Detention Centers at the Division FOB
• Areas Where the Final Product will be Disseminated

Base support companies such as International Management Services (IMS) provide translators and other services to coalition partners.  Their personnel possess widely varying education levels, and are generally willing to cooperate with MISO.  Open air café’s frequented by the local population are excellent places to interview local nationals.  The environment is permissive, and they are typically in the mood to socialize.  Local Airports are most valuable when full of LN passengers awaiting flights.  Developing a positive relationship with the airport manager will allow the MISO element to be privy to the departure/ arrival schedules.  FOB Entry control points are patronized by different TAs at different times.  A wise MISO element will observe these patterns and time their interviews appropriately.  LNs waiting for security checks at an ECP have very little to do, and are often quite happy to talk to MISO about any number of issues in order to pass the time.

Detention Centers require some planning and coordination to access.  For legal reasons MISO soldiers can only interact with inmates that are in a special transition status.  Those who wish to access these inmates must go through special training conducted by the detention facility.  Typically MISO will only be able to interview a handful of insurgent prisoners IOT test product, but their insights are often invaluable.  Tactical MISO Teams (TMTs) who request product have access to the precise TA where the product will ultimately be disseminated.  The feedback of the intended TA, more than any other, is the most reliable gauge as to how the populace will react to the MISO product.

Standard testing most closely resembles the process described in FM 3-05.301. Phase I, II, and III testing are all utilized in the standard process.  It generally takes a week or more to conduct all three phases, and it is labor intensive.  The standard process is best suited for high dollar value, high impact, products planned well in advance of actual distribution.

Abbreviated testing is the technique which is utilized the most in an austere counter-insurgency environment.  In abbreviated testing, phases I and III are utilized, but phase II is omitted.  In addition, the phase III TA is generally sampled by populations around the Division Headquarters which are most accessible to the Target Audience Analysis Team (TAAT) and or the Testing and Evaluation Team (TET) stationed there. 

In Afghanistan, the first group to get their message out is often perceived as the most credible.  If two groups have equal credibility, whichever message has the most solid foundation in Islam is considered the more authoritative.  Significant unforeseen developments drive battle drills which necessitate emergency testing.  For example on April 25, 2011 in Kandahar City 400 insurgent prisoners escaped from the Sarpoza maximum security prison.  This represented an immediate public safety threat that required a fast response from MISO.  The Standard and Abbreviated processes were not able to return testing results fast enough to get messaging out before the enemy.   

There is no set formula for emergency testing.  Sometimes there is enough time to throw together a quick mission to a local source.  In other instances there is only enough time to have a translator who is not on the MISO payroll take a quick look at the product and render his opinion.  Ultimately the Commander (CDR) must determine the level of risk he is willing to assume by disseminating product which has received inadequate testing.  The less testing performed, the greater the chance that the message is misunderstood, ineffective, or potentially offensive.  It is the professional responsibility of the Testing Group to inform the CDR as to what level of risk he is assuming by distributing a given product.

Evaluating Results

The process of evaluating pre-testing surveys requires pattern recognition, problem recognition and the development of a recommended course of action (COA). 

Pattern recognition is simple awareness of themes that develop during the testing process.  For instance if many people in the TA ask about an image then the image itself, or perhaps the concept it is paired with, may be unclear.

Problem recognition is the process of identifying and defining individual product elements which interfere with the TA’s ability to understand or accept the MISO message.  For instance, if the TA is visibly displeased by a particularly graphic image, then the issue is clear.  Other problems are not as easy to detect, such as those where the TA understands the message but simply does not accept the MISO argument.  Asking what text or imagery would better convey the intended MISO argument is an important follow-up question in such instances.  In the Afghanistan Regional Command South AO, 95% of all women are illiterate, and 75% of all men.  Products which rely on the ability to read, and do not target an educated subgroup in Afghan society, will fail. Dialectic differences between specific members of the TA and the translator also lead to misunderstanding. The interviewer must be on the look out for all of these potential confounds.

Results should be tabulated and inserted into a “Summary of Findings” such as those in Exhibit A1 & A2.  Care must be taken when presenting sample data points.  Stating that 100% of the TA accepted the MISO argument may be misleading if only two people were sampled!  It is better to state how many out of how many understood the message and or accepted the MISO argument.  For instance “30 out of 40 individuals polled understood the product to be about fighting crime”.

Recommendations should be based on the data that was collected during the pre-test, and should be designed to minimize the risk to the MISO Commander, while maximizing the effect of the MISO product.  If testing results show that the TA does not understand the product, or the TA is not moved to action by the product, then the product should be returned to the Product Development Detachment (PDD) for rework. The testing element should recommend products which test well for approval and distribution. It is important to note that positive test results do not necessarily mean the MISO product is completely optimized. Part of the testing process is to determine what changes will further enhance the message. The Product Summary should highlight all of these aspects.

Exhibit A1 Pretest Summary of Findings

Sample Methodology: Nonprobability Accidental Sample

Target Audience: b (Adult Afghanistan Populace)

Survey Sample: Afghan Populace

Pretest Location: ECP 5

DTG: 1 Mar13 0930

Product Number AFS10A04bRD10004

Questions Asked

1. Do you understand the meaning of this message? 31 of 31 individuals tested understood the message to be about why Coalition Forces are in Afghanistan.    

2. What does this message urge you to do? 26 of 31 individuals tested said the message urged them to be more understanding of CF, 25 of 31 individuals tested said the message did not urge them to do anything.

3. Is there anything offensive in this message? 31 of 31 individuals tested said there was nothing offensive in the message.

4. Do you have any suggestions to improve this message? 26 of 31 individuals tested suggested the message be shorter, 23 of 31 individuals tested had no suggestions, 22 of 31 individuals tested suggested not using anything in regards to 9/11.

Recommendations

This product did not effectively convey the intended PSYOP message with the majority of subjects polled.  While the tested group understood the meaning of the message, it was not well received.  Although a majority of the tested group stated that the message urged them to support the CF, the majority of the tested group thought the message was too long and needs to be shortened. Many in the tested group became disinterested in the message after the first 90 seconds.  Finally, there was a minority of the tested individuals that thought the use of the events of 9/11 was unnecessary and should be removed.  Testing recommends the message be shortened to 60 seconds.  If the events of 9/11 are to be used, they need to be explained more thoroughly, and put into context IOT adequately explain why the CF are here.

Exhibit A2 Pretest Summary of Findings

Sample Methodology: Nonprobability Quota Sample

Survey Sample: Phase I, Focus Group

Pretest Location: PSYOP Compound

DTG: 20APR13 1350

Product Number AFS11E03bDE10000

Questions Asked

5. In your words what does this image say? 3 of the 3 in the test group said it says giving charity during Zakat brings you closer to Allah.

6. What does this image urge you to do? 3 of the 3 in the test group said it urges them to help the poor and needy people.

7. What would you recommend for improving this image? 3 of the 3 in the test group said there are no changes necessary.

Sample Methodology: Nonprobability Accidental Sample

Survey Sample: Phase III, TA (Afghan people)

Pretest Location: Kandahar City Shop Owners

DTG: 21APR13 1300

Product Number AFS11E03bDE10000

Questions Asked

1. In your words what does this image say? 33 of the 33 in the tested groups said it is about Zakat a time for giving alms; they went on to talk about how Zakat is a pillar of Islam.

2. What does this image urge you to do? 33 of the 33 in the tested groups said to give to the poor and it is beautiful.

3. If you were given a sticker or water bottle with this symbol what would you think? 33 of the 33 in the tested groups said they would like rice on the sticker because it is very good for the Afghan people.

4. What suggestions do you have for improving this message? 33 of the 33 in the test group said there are no changes necessary.

Sample Methodology: Nonprobability Accidental Sample

Survey Sample: Phase III, TA (Afghan people)

Pretest Location: IMS Compound

DTG: 22APR13 1700

Product Number AFS11E03bDE10000

Questions Asked

1. What is your understanding of this message? 42 of the 42 in the tested groups said they understood it to be a message about giving to the poor on Zakat.

     2. What does this message urge you to do? 42 of the 42 in the tested groups said it urges them to give what they can in food and money to the poor and those who need it to survive, and those poor will use the money for good.

3. Do you find any part of this message offending, if so what part? 42 of the 42 in the tested groups said they found nothing offensive in the product.

4. What suggestions do you have for improving this message? 42 of the 42 in the tested groups said there are no changes necessary.

Recommendations

This message was well received by the sample population.  The message was clearly understood. The general populace thought the message to be positive and a good reminder during the time of Zakat. There are no recommendations for improvement of this product.

Post-Testing

Post-testing is a form of surveying which attempts to determine the effects of a specific MISO product on a TA.  This type of testing is conducted at key points within a MISO series.  The results support “decision points” to advance the series in question to the next level, or to maintain the status quo and continue shaping operations.  Shortage of MISO assets, combined with limited mission access, unfortunately makes ongoing post-testing impractical in many theatres of operation.  Every effort should be made to conduct a post-test of key products in a series by the time the series is complete. 

Unlike pre-testing, post-testing must be conducted with the actual target audience and therefore excludes phases I & II.  When choosing the location of the post-test it is important to pick regions where the product to be tested was disseminated.  While there is some limited movement of product from region to region as people move and communicate, by and large, people will be unable to comment on a product if it has not been distributed in their area.  Post-testing follows the same field guidelines for missions and inter-personal techniques as pre-testing.  Unlike pre-testing, post-testing questions tend to be fixed and experience less variation from product to product or survey to survey.  Typical questions are contained in Exhibit B, and are reproduced below.

• Have you seen/heard this product/message before?
• If you have seen/heard this product/message before, where?
• What do you understand the meaning of this message/product to be?
• What does this message/product urge you to do?
• Is there anything offensive in this message/product?
• What do you recommend for improving this message/product?

Exhibit B Post-test Survey       

Product Number:__________

Place (include grid): ___________   DTG: _________

Survey#: __________________     PAX in Group: _________

Gender:     M  F    Mix

Average Age:     5-9    10-14   15-19   20-29   30-39   40-49   50+

Occupation: ________________________

1. Have you seen/heard this product/message before?

2. If you have seen/heard this product/message before, where?

3. What do you understand the meaning of this message/product to be?

4. What does this message/product urge you to do?

5. Is there anything offensive in this message/product?

6. What do you recommend for improving this message/product?

Conclusion

In summation, product testing is a critical part of the MISO Tactical Development Detachment Process.  It validates the effectiveness of product, prevents embarrassing cultural misunderstandings, and is a required part of the MISO approval process.  Exhibits C1 & C2 contains flowcharts of the MISO pre-testing process for reference.

Those units which gloss over the testing process greatly increase the level of risk that they assume.  This risk may manifest itself in failed programs, broken relationships, or cultural misunderstanding which may increase sectarian violence or even drive insurgency.  Conversely, implementing the measures contained herein will significantly increase the effectiveness and efficiency of MISO product.  This in turn will catalyze the non-lethal targeting process and bolster supply economy.  The MISO resulting from a successful program will encourage enemies to surrender or defect, promote reconciliation, reduce inter-tribal conflict, increase freedom of movement, and support local governance and security efforts—in short it will provide tangible returns that any Battle Space Owner (BSO) will understand and appreciate.