By Harold Lowery, Carl Carhuff and Phillip Rowan


Abstract.

In this article we report our experiences applying Agile practices and Art of the Possible methods in a software sustainment organization, specifically the F-15 Avionics Integration Support Facility (AISF) at Robins Air Force Base.

Overview

Modern weapons are complex, software intensive systems. In Air Force terminology, the Operational Flight Program (OFP) software embedded in a weapon system allows for corrective, perfective, adaptive, and preventive changes to the system’s capabilities. [1] OFP sustainment ensures the embedded software continues to meet evolving requirements throughout its lifecycle. This article describes our experiences introducing some Agile practices along with Art of the Possible methods in the F-15 Avionics Integration Support Facility (AISF) at Robins Air Force Base.

Background

F-15 OFP

Both the F-15C/D air superiority fighter and F-15 Strike Eagle have federated system architectures in which multiple computers execute embedded software (OFP) and communicate via dedicated busses to accomplish the aircraft’s mission. The F-15 program office manages the development and release of OFPs as a “suite”, i.e. all OFPs are developed in parallel and release to the fleet together. In addition, aircraft capability upgrades and the integration of new weapons are coordinated with the suite cycle since such modifications nearly always require OFP updates. As a result, the OFP suite cycle is a high visibility, tightly managed set of inter-related projects that typically spans 3 – 5 years.

For the past 30 years the sustainment of most F-15 OFPs has been accomplished in the AISF at Robins AFB. Organizationally the AISF comprises a flight of three elements, which the authors have the privilege of supervising.

Avionics Element

The Avionics Element currently supports a large base of Ada, C, and assembly language code for various avionics sub-systems including two armament control computers that manage all weapons on the aircraft.

AISF Support Element

The Support Element performs all actions required to keep the AISF operational for OFP development, including the modification, maintenance, and repair of avionics and radar benches and their associated networks.

Radar Element

The Radar Element sustains the APG-63(v)1, APG-70, and APQ-180 radar systems. The latter is a APG-70 variant that flies on the AC-130U gunship. These systems were designed in the late 1980s to early 1990s, running JOVIAL and assembly language on proprietary hardware.

The Need for Change

The AISF sustains OFPs for six different avionics systems across multiple aircraft configurations with usually three suite cycles overlapping in different phases. Our relatively small workforce of some fifty engineers, computer scientists, and support staff are generally working on a dozen or more projects at any given moment. By the fall of 2014, as budget cuts constrained our ability to bring on additional staff, we began exploring ways to more efficiently manage the dozen or more projects we had in work at any given moment. We needed not only to ensure the most efficient utilization of our people, but also that we did not commit to more work than we could accomplish.

Agile

We found that the spirit of the Agile Manifesto, and the principles behind it [2], resonated with the “can do”, mission focused ethos of the AISF. While the requirements of the suite cycle process would not allow us to adopt Agile entirely, we felt we could advantageously adopt certain Agile/Scrum practices. In particular, we liked the emphasis on

1) continuously producing working code within time boxed sprints

2) frequent face to face communications among stakeholders

3) the use of burn-down charts for simple, focused metrics

4) addressing constraints in a timely manner

Art of the Possible

Also in 2014, the Air Force Sustainment Center (AFSC) commander, Lt Gen Bruce Litchfield, challenged the center to improve performance in every area by adopting best practices to achieve “Art of the Possible” (AoP) results. “The AFSC Way is a standard, repeatable tool set for achieving results. Disciplined and enlightened application of the AFSC Way enables the progression of improved performance from meeting expectation to achieving ‘Art of the Possible’”. [3] The “Art of the Possible” grew out of efforts to streamline aircraft Programmed Depot Maintenance (PDM) through incorporating lessons learned from Lean Production, Theory of Constraints, Six Sigma, etc.

Among the tenets of AoP is the reduction of work in progress through an emphasis on “flow”. On the PDM line this means continuously monitoring the progress of aircraft from one gate to the next. Any issue that threatens to reduce flow is identified and addressed as early as possible. Work in progress is strictly managed to prevent bottlenecks that impede flow.

First Steps

F-15 aircraft undergo PDM on a recurring 6-year cycle. Each year dozens of jets fly into Robins for an extensive schedule of inspection and maintenance actions performed by the 561st Aircraft Maintenance Squadron. The 561st has created an AoP “Production System” comprised of stages and gates representing the flow of aircraft through the PDM process. They have made significant progress in improving the flow of aircraft and returning them to service on schedule. As we considered how to integrate AoP concepts into our OFP sustainment process, we wanted to leverage their lessons learned. Late in 2014 we visited their weekly “Walk the Wall” meeting, so named because their entire gated process is mapped out on the conference room walls, showing detailed statistics on flow days, work in progress, etc, along with commentary – roughly 100 pieces of paper showing the status of every aircraft. The production system we saw that day was the squadron’s third revision to their implementation of AoP.

That visit sparked serious thought on how we could design and implement an AoP production system that delivered products within the F-15 block cycle constraints, all while incorporating Agile practices as integral building blocks.

Designing the Production System

To begin designing our production system, we held several sessions to map out our existing processes. One of the first questions we wrestled with was “What exactly should flow through our process?” This proved to be a surprisingly difficult question to answer, but after much discussion we settled on “tasks”. We defined a task to be something that had a deliverable and that a person could accomplish within a matter of days or at most a few weeks. From an Agile perspective this definition is unremarkable, but it stretched the boundaries of AoP, which had been mostly applied to physical items such as aircraft or their sub-systems.

In accordance with the basic principles of AoP, we decided to carefully track work in progress since “Controlling the amount of WIP into and within the process will ensure the process resources are not overwhelmed within the gates or within the system as a whole.” [4]

Since we had an existing requirement to release OFPs to flight test at the beginning of each month, it was an easy decision to time box our sprints to that cycle. We divided each monthly sprint into either 4 or 5 weeks as necessary to stay synchronized with the OFP release dates. With the AoP emphasis on “flow” we recognized the need for frequent feedback and decided to hold our Walk the Wall meetings weekly. Previously, status meetings had been monthly, but we knew quite well that the further into the future we asked our project leads to look, the hazier their crystal balls would become. This scheme of weekly review leading to a monthly release fits well with a key principle of Agile: “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” [5]

Tools

As we prepared to roll out our new process to our workforce, one issue that we struggled with was how to plan tasks and track actual performance. For years our parent organization, the 402d Software Maintenance Group (402 SMXG), had used a pair of home grown tools (SITE and EVTAT) in tandem to accomplish project planning, earned value accounting, and time and attendance accounting, however, the group was then in the process of transitioning to major upgrades of both tools. The uncertainty of when the new tool capabilities would be available and how they would be deployed made it difficult to integrate them with our new and evolving production system. Thus we elected to begin with a simple MS Excel spreadsheet to plan and track assigned tasks in a burn down chart.

From Theory to Practice

We launched the new production system and conducted our first sprint in February 2015.

Sprint Planning

In our production system, sprint planning occurs the week before the sprint begins. Project leads work with their team members to determine what tasks are to be worked, then use our standard Excel spreadsheet to create the product backlog. For each task they estimate in which week it will be completed. At the end of each week the project lead updates the burn down chart by simply entering the actual week a task starts and/or finishes.

An important point here is that every task is either not started, in work, or done. We never speak in terms of a task being “X % complete”. Our initial design did not account for unplanned tasks, but we soon learned we needed more flexibility. Now when unplanned tasks arise that must be worked in the current sprint, they are entered in the spread sheet as they occur, along with the estimate week of completion. Any tasks not completed in the current sprint are rolled over to the next sprint, along with unplanned tasks that are not urgent.

Walk the Wall (WtW)

The Walk the Wall meeting is held early the first day of the work week. Each team lead briefs 4 charts:

  1. Burn down chart
  2. What we did last week
  3. What will we do this week
  4. Constraints / Issues

The burndown chart gives the complete status of the team’s work. At a glance all participants can see the actual progress versus plan, the impact of unplanned tasks, and the amount of work in progress.

Figure 1 is a sample burn down chart showing the results of a full 4-week sprint. In this case, the team began the sprint committed to 16 tasks. They fell behind in week one and never quite recovered. Note the four unplanned tasks that occurred in weeks 1, 3, and 4. Work in progress spiked at five in week 3 and declined to the two unfinished tasks at the end of week 4, which will be carried over to the next sprint.

The second chart that is briefed lists the tasks that were accomplished during the previous week, while the third chart charts shows what tasks are to be worked in the current week. There is a strong emphasis on getting tasks done. A final chart lists any constraints that threaten to impede flow, as well as other supplementary information specific to the given project under consideration. Timely identification and resolution of constraints is an AoP method to prevent bottlenecks that reduce flow. “Any resource that is not available at the time and location necessary represents a constraint that must be identified, understood, communicated, elevated, and resolved appropriately.” [6]

Typically, about a dozen projects are briefed during the meeting, which usually lasts about 45 minutes. Often an issue will be addressed through an immediate decision. If not then the general rule is not to discuss it at length, but rather to assign the appropriate people to work it.

In the early sprints we emulated the aircraft WtW format by posting the charts on the conference room walls and having each team leader “walk” the room through his or her charts. The value of posting the charts on the conference room is that every project’s status is immediately available to anyone who walks into the room. However, to facilitate the presentation, we now project them on a large screen during our meetings, in addition to posting slides on the walls.

Results

We have found the chief benefit of a weekly Walk the Wall review is that it allows us to surface and address issues quickly. Communication between individual teams that support each other is enhanced when the leads are in the same room and working off the same metrics.

Each project’s briefing supports that communication by a laser focus on the results. The burn down chart provides a clear, visual summary of progress being made, work in progress, and unplanned tasks.

What’s Next?

At this point we have developed practices and metrics that allow us, as first line supervisors, to manage projects more effectively than before, with an emphasis on identifying and resolving issues that threaten flow. After reviewing these results, our senior leadership said “Okay, what you’ve done works at the ‘tactical’, day by day level. What metrics can you collect that will allow your boss and his boss to manage at a ‘strategic’ level?”

So our next action is to look for practices and metrics that support reporting to and decision making by higher levels of leadership.

Figures and Tables:

Figure 1: Example Burndown Chart ( Click to view image )


References and Notes

1. Guidelines for Successful Acquisition and Management of Software-Intensive Systems: Weapon Systems, Command and Control Systems, Management Information Systems - Condensed Version, Chapter 16, 4.0 February 2003

2. http://www.agilemanifesto.org/principles.html

3. Art of the Possible (AFD-140911-029), Air Force Sustainment Center, 2014

4. Ibid, page 48

5. Beck, Kent; et al. (2001). “Principles behind the Agile Manifesto”. Agile Alliance

6. Art of the Possible (AFD-140911-029), Air Force Sustainment Center, 2014, page 49


Harold Lowery

Click to view image

Harold Lowery is the Director of Radar OFP element, 578th Software Maintenance Support Squadron. Most of his 34-year career with the U. S. Air Force has been devoted to the development and support of the F-15 air superiority fighter, with occasional excursions into other DoD weapon systems. He holds a BS and MS from the University of Georgia – Go Dawgs.

Carl Carhuff

Click to view image

Carl “Pnut” Carhuff is the Director of the 578 SMXS, Flight B, Element A, responsible for F-15 Avionics OFP Development and OFP Test and Integration. He is a retired F-15C Pilot and has been in Civil Service for 6 years working in F-15 Software development the entire time as the OFP Suite Lead and now as Element director. He received a BS in Electrical Engineering from the USAF Academy, and an MS in Aviation management from Embry Riddle University. He is a diehard New York Yankees fan and an avid golfer.

Phillip Rowan

Click to view image

Phillip Rowan is the Director of the Avionics Integration Support Facility, 578th Software Maintenance Squadron. During his 11-year career with the U.S Air Force he has supported Test Program Set development, continuous process improvement initiatives, and software support for the F-15. He holds a MS from Mercer University, however he considers himself a Tennessee Volunteer from his undergraduate work obtaining a BS in Computer Engineering.


« Previous Next »