Enhancing Red Team Performance: Driving Measurable Value and Quality Outcomes with Process Improvement

Summary

Defense and Intelligence Community (IC) Red Team, Red Cell and Devil’s Advocate capabilities continue to improve technically and methodically at the practitioner level and within small adaptable groups. The “Red” element can enhance analytical products, challenge critical planning outcomes, scrutinize the viability of source reporting, and assess potential attacks or responses.  Unfortunately, cases of leadership’s lack of support, organizational cultures, and how the “Red” capability is viewed or defined, may be hindering Red Teams and their support to decision makers and internal customers.  DOD and IC leaders could benefit more from this alternative view or testing capability with process improvements that identify and resolve constraints and define value or capacity in the Red Team service offering.  With a growing need for intelligence and operational supremacy in a resource constricted defense environment, commercial Lean Six Sigma concepts may help manage scarce Red Team resources effectively while boosting its analytical production and penetration testing capabilities for customer needs and DOD and IC value add.

Introduction

Red Teams (or cells) and Red Teaming alternative view thought processes have long been used as tools by the leadership of government and commercial enterprises with a purpose of reducing risk, improving planning, and increasing opportunities through additional rigor (Defense Science Board, 2003).  The associated Red Team names and purposes, however, are not the same despite being attributed to alternative view or approach needs to reduce recognized biased personal view potentials, or “mind-set”, that could otherwise limit wider considerations to a topic.  As a decision support planning tool, Red Teaming puts members into a “red” opposition force mind-set to consider the motives, intent, tactics, techniques and procedures that a potential adversary might use against friendly capabilities (McGannon and Pollick).   

Red Team members ask questions to understand decision issues and to gain a “sense of self” from alternative outside-in views while thinking within the constructs of the cultures they are examining to identify the problem and arrive at a better solution from the opponents perspective (Craig, 2007).  

Red Team Functionality

The Red Teaming concept evolved from roots in military science to help strategic planners emulate an adversary in a structured and iterative process--executed by trained, educated, and practiced team members with access to relevant subject matter expertise (Gallegos and Smith, 2006; Fontenot, 2005).    When used correctly, a Red Team can provide straightforward information to inform and shape adaptations in an otherwise flawed plan about actual, proven vulnerabilities and find several weaknesses.  The outcome can drive improvements, guide the resourcing of contingency plans, and posture the force for an array of possible outcomes. 

According to an institutional examination of Red Team use, the Red Team practice is not [always] done very well and there are formidable challenges in the form of independence, interaction, and purpose that cause actual and perceived failures and contribute to issues in establishing and sustaining effective Red Teams and associated Red Teaming processes (Defense Science Board, 2003). 

In the case of a Mandatory Declassification Review government document "Misreading Intentions: Iraq's Reaction to Inspection Created Picture of Deception" (Jan 5, 2006), the CIA allegedly blamed "analyst liabilities" for failures to assess Iraq’s virtually non-existent WMD capabilities. Liabilities were defined as neglecting to examine Iraq's deceptive behavior "through an Iraqi prism".    

The CIA’s released Occasional Paper, “Rethinking ‘Alternative Analysis’ to Address Transnational Threats”, an internal agency work-group questioned the Directorate of Intelligence’s (DI) focus on modeling itself after the academic realm.  The work-group challenged the DI’s “narrow academic” focus on getting data and facts right as an endpoint as opposed to using those outcomes as analytical starting points to build more questions that challenge assumptions.

Today’s Red Team Challenges for Success

The Red Team role may have been mandated by government administration post-9/11.  In an article by Foreign Policy magazine’s Mark Perry published in 2010 on CENTCOM’s Red Teams, the author claims that “after the 9/11 attacks, every U.S. intelligence agency was mandated to have a Red Team.  Specifically, the CENTCOM unit was established in April 2006 following an order by then-Secretary of Defense Donald Rumsfeld, with a charter to provide the CENTCOM commander, leadership, and staff with alternative viewpoints, challenge common assumptions, and anticipate unintended consequences of events and actions.” 

Despite mandates, issues remain today within areas of the defense and intelligence community regarding formally established Red Team roles and inclusion, the holistic Red Teaming process system, and deliverable quality variations which can reduce decision support trust and the value to both the consumer and stakeholder (Defense Science Board, 2003; British Defence, 2010; DHS, 2007). 

Unclassified research for this writing did not present any locatable overarching guidance on the practice, process, and interagency collaboration of Red Teams and how they should be created, run and integrated within the IC and DOD.  Data pointing to the fact that some entities have a Cell, a Team, a Devil’s Advocate practice, a club, and interest group, validates that there is minimal over-arching guidance, or it at least is not widely enforced and generally accepted.

Anecdotal evidence gained by informal conversations across the community suggests that across the Federal community, Red Teams can be ostracized and shunned due to their perceived role in “pointing out faults”, “incessantly tagging improvements”, and “creating unwanted angst in departments”.  It is for this reason that some organizations use ad hoc Red Team analysts or actors to assuage concerns of a full-time career damaging duty that rarely gains reward and recognition.  

However, these issues tend to be more implied than explicit, meaning end-user customer and stakeholder statements of Red Team wants or desires are more generalized than specific regarding grievances of problems, difficulties and dissatisfactions.  This ambiguity creates a challenge in balancing the seriousness of a potential problem and how comprehensive—or how costly— the solution “fix” should be to provide better planning, analysis, and vulnerability safeguarding value-adds.  

Common Red Team Issues

The most common issues that have emerged in discussions with Red Teams and the communities they interact with involve the Red Team’s Culture, Organizational Patterns of Comfort, and New Skill Demands.

Red Team Culture- The formal Red Team unit and its immediate leadership are often comprised of imaginative unconventional thinking individuals that prefer not to be bounded by rigid guidelines or by the need for evidence and corroboration.  For some Red Teams, the perception of constraint by adopting mandated conventional rigid processes may cause the team to resist adoption for fear that it will inhibit some aspects of thought freedom that are considered vital to the Red Team’s resourceful thinking.  On the other hand, with no proof that process impedes effective Red Team analysis, actual unrestricted Red Team thought and the illusion of unrestricted Red Team thought could be a potential fallacy within the myth and reality of RT shortcomings or value-add benefits. 

Patterns of Comfort-   A lack of understanding of how to properly use a Red Team beyond normal comfort is an issue.  Leadership can find more comfort and trust working with their current networks despite the fact that the mindset will likely be quite similar, thus nullifying much alternative thought.  According to Richards Heuer’s Psychology of Intelligence Analysis, proven human bias and cognitive challenges fall to challenges of information being most accepted when it confirms already held judgments. Heuer revalidates the institutional challenge point stating, “New ideas must pass over a number of hurdles before it is embraced as an organizational product.”  Former CIA Analyst, Morgan Jones, describes in her book The Thinker’s Toolkit, the flawed and less effective approach of surrounding yourself with like-minded thinkers can produce the analytical missteps of a closed mind, satisficing methods, and alternatives that are not considered. 

New Skill Demands-   The issue of new skill demands occurs when a Red Team is simply not able to mentally get inside of the enemy's decision cycle or darker mindsets and have not achieved the ability to understand how a particular adversary thinks.  In the field of non-state actors and intrastate violence, Red Teams may need to move beyond current problem solving capabilities or human reasoning skill sets and consider the field of psycho-socio-cultural profiling to better assess mental and emotional capacity of the target or populace.  The move would bring Red Teams closer to Conflict Analysis in a combination of military science, cross-cultural psychoanalytical anthropology, and emerging society studies, to predict the behavior of non-state actors and communities in key situations (Christian, 2012). 

If Red Teams could harness these issues and take from proven methodologies and practitioner improvements to bring a more comprehensive value-driven approach to the organizations they support, it may create a more formalized role for the profession.  It could also assist in alleviating some of the typical Red Team challenges such as gaining more access, securing leadership’s engagement, and interacting with leadership that is committed to making changes or keeping an open mind to alternatives based on Red Team findings (Craig, 2007).  

In other cases, it may make more sense to push the Red Team role to a neutral external entity that could mitigate an internal Red Team’s worry of career and fears of the current Reduction in Force (RIF) environment based on providing unbiased and unvarnished deliverables that may create unwanted repercussions. Ultimately, it becomes an organization and leadership driven issue that can be improved with more rigorous problem identification or definition, adept process enhancement and calibrated change management.

Red Team Improvement Efforts

Leadership that creates high reliability organizations can consider red teaming as a way to posture the force for a sustainable competitive advantage by capturing the value and placement of a critical alternate view capability needed today as we approach the next generation of indirect and direct approaches globally for influence, outcomes, and building relationships.

Best Practices in Training

At present, there are a handful of core frameworks, trainings, and champions that have been recognized or are generally accepted in the Red Teaming community as high reliability best practices.  Sandia National Laboratories, MITRE, and the University of Foreign Military and Cultural Studies (UFMCS) have made extensive contributions to the Red Team community and continue to produce or improve Red Team practitioners and leaders.   

The University of Foreign Military and Cultural Studies has, among other things, leveraged their higher learning environment and global experience with socio-cultural and anthropological expertise to better prepare their practitioners with improved mindset of foreign actors and critical thinking skills.   

The methodologies and educational learning outcomes from these organizations provide specific steps, phases, and checklists to organize, tailor and conduct more effective Red Teaming activities in penetration testing, modeling and simulation, and analysis. 

New Domains for Red Teaming Training

Despite the Red Team training resources available to soldiers and government analysts, a number of entities within the military and agency parent structures still lack the consistency in training Red Team members in proper application and utilities of function, and do not adhere to a formalized doctrine to produce consistent results.  In other cases, it may be that the course of action plans and methods of achieving tactical outcomes or strategies may be outside of the Red Teams’ expertise and purview---and outside of leadership’s knowledgebase.  Emerging Red Team doctrine areas of Cyber and Counter Threat Finance require specific capabilities and expertise.  These two tools of the battle space should be considered in future inclusion and support.

Indirect Effects

Leaders in defense and intelligence are coping with perceptible and imperceptible elements in conflicts and relationship shaping across political, economic, social, information, and security environments.  Adversaries are learning and adapting faster than U.S. institutions (IC, DOD, LE, etc.), and are leveraging asymmetric and hybrid strategies to offset U.S. technical and numerical superiority.  One has to get in front of these issues by understanding dynamically changing human factors.

Add to this challenge, courses of action have to consider a wide range of potentially conflicting agendas within the same government where there may be disconnects in interagency planning and objectives.  By widening the aperture of the possible, Red Teams can help leaders consider the range of links that converge policy, authorities, strategy, tactics, and disparate whole of government aims.  As defense and intelligence look to achieve more indirect effects in strategic warfare, alternative analysis will be paramount to identifying nuanced subtleties that can be evaluated or calculated against. 

Demonstrating Value

In a short case study of the Canadian Forces (CF), the CF approach to Red Teaming has been manifested by ad hoc last minute assignments of individuals who are chartered at the time to role-play the adversary (Lauder).  This is not uncommon to some of our own Red Teaming capabilities.  In the Canadian Army Journal, the Red Teaming failure is described as occurring when the activities are applied in a largely haphazard and casual fashion, and a strategy to formalize the concept, and professionalized the activity for application in training, planning, and operational environments, has been absent (Lauder, 2009).    

Indeed, a population of Red Teams and practitioners are attempting to improve their skills, but in organizations where the Red Team customer is unhappy, there is likely a significant service or organizational failure of the overall dynamics and processes in place for Red Team utilization, and as a result, the end products and Red Team may fail to do what they are designed to do for leaders.

Leadership Support

Few Red Teams in the IC and DOD community can claim that they are consistently embraced by their organization in a prioritized, supported, and enabled manner.   Conditions exist for the use and benefit of Red Teams, but leadership often has not endorsed the environmental approval and guidance for Red Teams to be most effective to their organization.

In general, organizations that regularly assess and demonstrate valuable internal service offerings typically gain leadership’s priority, support, sustainment and enablement in the day to day operations of those services.  This enterprise-driven value commitment plays an integral part to functional standards requirements and acceptance by internal cultures to mirror leadership’s policies and procedures of processes---and their sustainment over time. 

Required Conditions

In the UK’s Guide to Red Teaming, created by the Development, Concepts, and Doctrine Centre of the Ministry of Defence, the leadership approved guidance note specifies a number of conditions required for effective Red Teaming. 

Among the conditions are:

·       Environments that tolerate and value internal criticism and challenge;

·       Essential top-cover and commander’s confidence to assert findings and be seriously considered;

·       Engagement to interact in a non-competitive manner with other planning teams to create more focused solutions and insights;

·       Full appreciation and understanding of the actual issues at hand (ex. Appropriate read-on to the specific mission or program);

·       Advanced engagement with operational, analytic, or planning teams before major problems arise to better anticipate issues; and

·       Composition of suitably expert and experienced staff for the job, to include the capability to reach out to broader organizations, academia, institutions, and industry to supplement subject matter experts as required.

The guide further asserts that lacking of these conditions and in cases where Red Teams are removed from the decision making process, suffer from a dearth of adequate organizational interactions, and are not supported by the commander (or is unduly influenced by him), success will be unlikely.

The U.S. Marine Corps similarly supports this position in their MSTP Pamphlet Red Cell-Green Cell by emphasizing the need of Red Teams so a commander can better understand and thoroughly plan proper courses of action (COA), and to have Red Teams involved directly at the same time as the operational planning team (OPT) with proper resources to assist with proper Problem Framing, Coordinating, and Organizing (USMC, 2011).

Reducing the Red Team’s Organizational Vulnerability

The UK MoD and USMC doctrinal publications at least address the issue of establishing appropriate frameworks and relationships to ensure the team and therefore the greater organization is successful.  Still, measures of success remain elusive, which can leave a Red Team program vulnerable to defining where they provide value to planning, analysis, and assessments in a more tangible way to their organization and the community. 

If Red Teams have been mandated across DoD, then perhaps an entity like the Under Secretary for Intelligence, or USD(I), could harness a multi-agency WoG Community of Purpose (CoP) that would not impose regulations but could provide recommendations and support.  Best practices could be identified across the IC and DoD to serve in mentor protégé capacities, and the various roles of Red Team alternative analysis or penetration testing could be standardized to improve understanding of capabilities and limitations.  This would at least serve as a baseline starting point that would set a stage for metrics and value proposals. 

Measuring Effectiveness for Customer Value Enhancement

Many things can be measured qualitatively or quantitatively to include a service offering such as Red Team assessments and analysis.  Sandia National Laboratories partnered with the University of Wisconsin-Madison in 2004 and even offers a course on Red Team Metrics.  UFMCS covers aspects of this as well in their Red Team leadership courses.   

It was stated in the U of W-Madison study that while adversary metrics are associated with Red Team characterization, including progression of effort, they are not focused upon the performance of Red Team characterization, including progression of efforts, and they are not focused upon the performance of the Red Team itself (2004). 

While some courses teach measures of effectiveness to include experience, composition, process, capability, and knowledge, the study claimed that nothing measured effectiveness of a particular Red Team or Red Team simulation before, during, and after a Red Teaming effort (U of W-Madison, 2004). 

Some measures are displayed below (in no particular order or relationship to one another).

Red Teams can improve upon these performance areas with custom designed metrics appropriate to output, outcome, and impact indicators to augment accountability, compliance, and mission or analytical results.  When tied to statistical analysis, valid measures of performance and impact can then be transformed into actionable information to stakeholders/leadership.

Improving Red Teams Processes Organizationally

Organizations that see the value of Red Teams but are experiencing greater institutional service failures may need to implement a formal decision support process improvement initiative to create a higher degree of evidence based decisions and results.  One example of endorsing a more comprehensive approach to change is the Lean Six Sigma (LSS) approach.  LSS is most notable in commercial industries for quality control and gaining the support of leadership to instill a cultural change in the adoption of process improvement implementation.

The use of such an approach can be held lightly or implemented formally.  Since human capital and their working environments are complex entities, oversimplified ways to deal with organizational issues often do not persist effectively or with desired longevity.  The Red Team capability, parent component, and its stakeholders assessed in a LSS approach could identify symptoms and causes of inefficiencies and internal issues to remove “Waste” of time, efforts, and resources that contribute to the failure of meeting customer or leadership’s acceptance criteria, and be more sustainable over time. 

A customized representation of a defense-related Red Team capability enveloped by a customer-focused pursuit of perfection has been developed below to show the relationships of the internal mechanisms and overarching considerations in creating and sustaining shareholder value and customer focus. 

Ultimately, a process improvement solution can establish principles as a guide to address customers, value, measurement, pursuit of perfection, and assistance with change management efforts to ensure improvements are implemented effectively with consideration to culture and priorities.

Controversy of Measuring Intangibles

Given the choice, most organizations and individuals would prefer not to be measured.  It is feasible that LSS as a tool may not even have a proper place for the topic of Red Team improvement.  Some may argue that Red Team value comes in the form of the “Insights” it can provide.  Information and therefore insights are items that, indeed, cannot be easily measured.

Information itself is often valueless unless it can be translated into usable knowledge.  It is a common statement within the Intelligence and Defense community that Information has no value unless it can be turned into Intelligence, and that Intelligence is of no value unless it is actionable or insightful. 

In the context of a Red Team’s value measurement of Insights, value could be placed on time saved in re-planning (upon learning that objectives were not achievable) or the opportunity cost of resources and lives that could be lost with a bad plan.  But since the insights that a Red Team delivers to provide stakeholders and customers is one piece of the decision making process, its value may never be recognized in isolation as a critical stand-out component.

Technical Measurement of Insights and Lean Six Sigma

Quantitatively, Red Team “Insights” value could be weighted according to the observations that reduce uncertainty where the result can be determined as an amount and is proportional to demand versus availability.  The key word is Demand.  A Red Team that provides insights may be providing something that is in demand.  The true value of demanded information is accepted in general terms. The choice and measurement that can equate to Red Team availability, budget, convenience, cost, efficiency, evaluation and determination of value added is based on the perceptions of the user.  Information has value, and even more so when it becomes useful knowledge to the individual receiving it. The determination of its value and the choice of payment or nonpayment is a final decision made by the user and not by the Red Team.

Conclusion

Global threats are increasingly complicated to identify, define, and mitigate.  In particular, it is critical to assess with the proper analysis and methods to engage and influence a random actor, follow money, predict the attack of a cell of autonomous planners and violent actors, or understand a traumatized community for peace operations and humanitarian missions.  At present, government and military institutions are challenged with aspects of Plans and Project Support, Mission Analysis, Alternative Studies, and Threat Emulation, but they have the Red Team capability in their future arsenal.  If the commander or department lead is not using the Red Team, alternative thought resources may not be a perceived value to the decisions they make, which is a significant vulnerability to strategic and tactical planning in government and military organizations.  That leadership shortcoming becomes part of the greater DOD and IC burden to fix. 

Red Teams are trying to meet these challenges while adding new complexities to their own operating environment.  New tools are being implemented in these environments to create scenarios, develop indicators, monitor trends, and evaluate comingled industries, political movements, and social developments that could influence outcomes that may otherwise derail planning and analysis if not considered.  The planning and analytical departments that support enhanced Red Team capabilities will likely feel associated pressure for evidence-based accountability and budgetary resource validation in the near future when new expenses are incurred. 

Regardless of whether a military or government entity chooses to implement evidence based performance measures or a Lean Six Sigma process to improve their Red Team’s effectiveness or not, transformational change should occur within Red Team organizations that are operating in an ad hoc or ineffective manner if leadership is not pleased with their services and products.  The issues, as stated, are repairable and with examination and changes can create greater efficiencies and value improvements within the Red Team and between the leaders they support.

Resources

Carayon, P. and Kraemer, S., “Red Team Performance: Summary of Findings; University of Wisconsin-Madison & IDART: Sandia National Laboratories,” June 2004, University of Wisconsin Center for Quality and Productivity Improvement.

Christian, P., “Engaging Traumatized Communities in Village Stability Operations”, Summer-Fall 2012, Department of African Studies, National Intelligence University.

Craig, S., Reflections from a Red Team Leader, Military Review, March-April 2007

Department of Homeland Security, Homeland Security Exercise and Evaluation Program, Volume II: Exercise Planning and Conduct, February 2007.

Duggan, D.; Hutchinson, R, “Red Teaming 101,” Sandia National Laboratories, 17 July 2004, www.cs.nmt.edu/%7Ecs491_02/RedTeaming-4hr.pdf

Fontenot, G., “Seeing Red: Creating a Red-Team Capability for Blue Force,” Military Review (September-October 2005): 4-8.

Gallegos, F. and Smith, M., Red Teams: An Audit Tool, Technique and Methodology for Information Assurance, ISACA Journal Online, 2006, www.isaca.org

Gladman, B.W., “The ‘Best Practices’ of Red Teaming,” DRDC CORA TM 2007-29, Centre for Operational Research and Analysis (2007)

Lauder, M., Red Dawn: The Emergence of a Red Teaming Capability in the Canadian Forces, Canadian Army Journal Vol. 12.2 Summer 2009

McGannon, M. and Pollick, R., AFRL RED TEAM COOKBOOK, Volume 1: Red Teaming 101, Plans and Programs Directorate (AFRL/XP), Air Force Research Laboratory, Air Force Materiel Command

MITRE, Defense-Information Assurance Red Team, June 2000

http://www.mitre.org/news/digest/archives/2000/defense_red_team.html, Accessed August 24, 2012

Sandia Labs’ Information Design Assurance Red Team (IDART), 2009, http://idart.sandia.gov/ (accessed August 24 2012).

Skroch, M., Modeling and Simulation of Red Teaming Part 1: Why Red Team M&S?, Sandia National Laboratories, 2 November 2009 – Rev 3, published by Red Team Journal, redteamjournal.com.  

University of Foreign Military and Cultural Studies, Red Team Handbook v. 5, 15 April 2011

U.S. Department of Defense, Defense Science Board Task Force, “The Role and Status of DoD Red Teaming Activities,” September 2003, www.acq.osd.mil/dsb/redteam.pdf

U.S. Marine Corps, Marine Air Ground Task Force Staff Training Program (MSTP) Pamphlet 2-0.1, Red Cell - Green Cell, October 2011.