18F

Making government simple, beautiful, and easy to use.

Say Hello to the New Presidential Innovation Fellows

The third class of Presidential Innovation Fellows was sworn into duty last week by former CTO Todd Park. We are excited to welcome them into the 18F family.

We recently welcomed the newest group of Presidential Innovation Fellows into the federal government. This diverse group represents some of the nation’s most talented and creative civic-minded innovators.

More than a thousand candidates applied to serve the country in this unique capacity. From this pool of amazing and incredibly motivated applicants, we selected almost 30 designers, developers, entrepreneurs and executives to bring their unique skills into government.

Read the full post at the GSA blog.

Meet the 27 Fellows.

Read more about the projects that make up these initiatives, and the previous successes the program has helped shape.

The Encasement Strategy: On Legacy Systems and the Importance of APIs

By Robert L. Read, PhD with illustrations by Michelle Hertzfeld

In 1986 a nuclear reactor known as Chernobyl released harmful radioactivity which spread over much of the western USSR and Europe. The core of this reactor remains a glowing, ineradicable mass of deadly radioactive lava in the middle of a large Exclusion Zone unfit for human habitation.

The Chernobyl reactor core could not be removed. It was and is too big, too hot, and there is no where for it to go. Instead, it was entombed in a concrete sarcophagus where it will radiate harmlessly for decades if not centuries. This was unfortunately the best outcome that could be achieved. In the software industry, I’ve often seen this same approach used with legacy software systems. I call it The Encasement Strategy.

Legacy systems make everyone that has to touch them queasy, from the software engineers to the managers. But most especially there comes a time when the system no longer serves the most important constituent of all, the customer. This not only contributes to inefficiency, but can sometimes have detrimental effects on the users of a system.

legacy software can be toxic to customers and stakeholders

Like Chernobyl, these systems are toxic; but unlike a power plant, what they once produced is not fungible with other sources. There is often no replacement for the legacy system.

The basic mechanism for remediating such a system comes straight out of Computer Programming 101: you create a well-designed interface. In modern terms, this is an Application Programming Interface or API. That term API used to be more general, but now it almost always connotes a web-based interface accessed through HTTP and usually using JSON as its data format.

An API is an inter-face, a face, a façade or a wall between. It allows the user blissful ignorance of what precisely is behind the wall. You need only worry about what comes and goes through the gate. What lies beyond—whether it’s magic, or a red-hot mass of legacy code—is no longer the user’s concern.The customers on the user side of the API are protected from the toxins, leaving the engineers to deal with implementation.

There is something magical about this basic act of defining an interface. To paraphrase Buckminster Fuller, to define is divine. It creates something simple and understandable from nothing, something you can grab onto, something solid.

By defining an API, you can begin to immediately serve the customer, because you can build a modern GUI on top of it, unencumbered by the legacy of the past. You can begin to build what history has shown needs to be built to serve the customer, whether that is the US citizen at large, or a division of an office, or a bureau of an agency. You may choose to allow outsiders to directly make calls to your API, but even if you do not do this, you can create independence of the legacy technology.

Sometimes, such an API can be constructed based on a clear engineering understanding of the internals of the legacy system. This is the best approach; however, it may be impractical if the knowledge and understanding of the system has been lost. In such a case, programmers are wont to resort to reverse engineering solutions, as I did recently. Any system which offers a GUI (graphical user interface) to users can be reverse engineered to construct a programmatic interface on top of that interface. We generally call this scraping the GUI, and it isn’t pretty. It leads to the absurd architectural diagram of a GUI on top of an API on top of a GUI on top of a miasma. But it gets the job done, and that is what a pragmatic software engineer must care about: serving the customer.

build an API and an interface to scrape information from a legacy system

Once a valuable API is defined, there is a wall between decisions about how to effectively use the API that completely divorces them from decisions about what to do with the code that implements the API. Efforts to rewrite the legacy system may proceed mostly independent of the efforts to build functionality that uses the API. Or, efforts to rewrite it may not proceed at all—the Encasement Strategy.

As an engineer who loves hard problems, the idea of leaving a legacy system in place and not attempting to rewrite it is a serious challenge. But I think we should always make that decision independent of decisions on how to best serve the customer.

Let us work through a highly contrived thought experiment. Imagine that 100 years from now there is a team of 5 highly skilled specialists, known as Software Conservationists. Their sole job is to maintain the sealed-off core of the legacy system which is STILL implementing an API that serves the US Citizen. Just like Art Conservationists working at the Smithsonian today, future citizens could train to do it.

The Software Conservationists are employed because of two decisions made today:

  1. a decision to create an API and then—
  2. a decision not to rewrite the legacy system.

Let us say that it costs $1,000,000 present-day dollars to maintain this team every year from now until the corium in Chernobyl is no longer hot.

How much money would you have to save this year in order to justify paying out an annuity of $1,000,000? Assuming that one could obtain a risk-free 3% after-inflation return on an investment (or, in accounting terms, discount rate of 3%), how much money would you have to save to justify making a decision today that creates the Software Conservationists profession a century from now? The answer is an elementary present value calculation: $34 million or more.

If a realist who is keeping her users top of mind can save the US Taxpayer $34 million today, she should employ the Encasement Strategy and not be ashamed of it. Such a realist should of course recognize the long-term effects of Software Conservation versus the creation of a new, modern system.

Whether the details of your toxic system lead you to begin the legacy rewrite immediately or to employ the Encasement Strategy of delaying the rewrite indefinitely, get started on a well-designed API today.

A New Look at the Freedom of Information Act

By Jackie Kazil, Shashank Khandelwal, Raphael Majma, Eric Mill, and Victor Diaz Zapanta

There are many ways the public can get information from the Federal Government. For example, you can check out data.gov to find scores of datasets and APIs, agency websites for information about their work, or other important information in online FOIA Libraries.

Or you can also just ask for it.

Since 1966, the Freedom of Information Act, FOIA, has granted the public the right to access information from the Federal Government. This public right has been maintained for decades and has served as the backbone for information disclosures. This has led to the publication of many important news stories and greater public awareness around government activities.

As demand for information continues to grow, it is important to continue iterating the ways we refine the FOIA request process. Our effort is one of a number of commitments towards creating a more open, transparent government. We will explore how to supplement the work that has already been done by creating tools to improve the online FOIA requests process by designing for the user.

photo: a mockup of our app on an iPhone
Above: an illustrative prototype running on a mobile device (the logo in the photo is not a live URL)

We are exploring building tools that:

  • improve the FOIA request submission experience;
  • create a scalable infrastructure for making requests to federal agencies; and
  • make it easier for requesters to find records and other information that have already been made available online.

This effort will be conducted with the assistance of a number of agencies and offices within the Federal Government. A FOIA Task Force, which consists of representatives from the Department of Justice, Environmental Protection Agency, the Office of Management and Budget, the Office of Science and Technology Policy, and others, has been created to oversee the creation of these open source software resources.

To reach our goals, the FOIA team at 18F has been meeting with stakeholders, both inside and outside the government, to discuss some of the practical obstacles impeding the current FOIA experience.

As we continue, we look forward to informing you of what we learned, but more importantly we look forward to informing you of what we’re building. We currently have a prototype available of what a consolidated request submission hub could look like. Please follow along at our main FOIA repo, give us feedback or contribute, and look for more updates in the future.

Creating an Open FEC

By Raphael Majma, Sean Herron, Noah Manger, Victor Diaz Zapanta, and Amos Stone

Campaign finance information is not very approachable, even when made available as open data. The laws that regulate how money can be spent around elections are important to our democracy, but sometimes it’s difficult to understand how these laws apply. Between Senate, House, and Presidential campaigns, thousands of people run for office on a regular basis (every two years for the House of Representatives, every six years for the Senate, and every four years for the Presidency). With each election comes a huge collection of information on candidates and political committees, most notably the contributions and expenditures they receive and make. This information can, at times, be difficult to understand, especially without a full understanding of the context of the rules and regulations around how it is collected and monitored.

The good news is that this information is already made available for public use by the Federal Election Commission (FEC). The FEC is a regulatory agency that, among other things, releases campaign finance information. The commission is bipartisan and has long shown a commitment to making sure their actions fairly and equitably disclose important data around certain election activities. This data powers important work by a number of transparency groups, is used by journalists to report on election trends, and informs the public about how money is spent around federal elections.

A few weeks ago, FEC and 18F started to explore how all of this great information can be better presented to the public. Over the past few weeks, we’ve begun learning all we can about the FEC, the process by which it collects and shares data, and how individuals outside of FEC use that data on a regular basis to gain insights into the workings of our democracy.

Users of the FEC website vary in profession and expertise in campaign finance policy and laws, which leads us to consider:

  • different formats and presentations
  • interactive web presentation or aggregated data as CSV reports
  • raw filing data for independent analysis

Creating a solution that will be effective for different use cases is a challenge — exactly the type of challenge that fits what 18F was created for. We’re excited about the opportunity to work on a project of such critical importance to the United States.

We’re still learning about the FEC’s current and future users and will continue working with groups both inside and outside FEC to better understand their needs. We are performing research to better understand FEC’s current audience, as well as the audience they’re not currently reaching. We will test the strengths and weaknesses of our ideas and understanding of the FEC’s website users by developing small-scale prototypes we think will address user needs then let people use and critique these prototypes.

This work, and your input, will directly impact how we and the FEC reimagine their digital presence. Please follow our repository and send us specific requests or ideas through our issue tracker. You can also be bold and provide us a specific approach or implementation idea through a pull request! We can’t wait to work with you to create a more elegant, user friendly, and accessible way to interact with the FEC.

The Contributor’s Guide to 18F: Code for the Common Good

By Robert L. Read, PhD

Introduction

Transparency in coding makes code more secure. Open-source development is development in the light, sometimes a harsh light, that shows every blemish. At 18F we strongly believe this improves the rapidity of our coding and the quality and security of the code.

We keep the code open to each other, which allows us to quickly scrub in on projects and to dexterously apply the most talented resources to a problem without too much concern for who is formally working on or in charge of a given project. The code is also open to Federal employees from outside 18F and from other agencies. This means that they may both review the code, offers suggestions, and, in some cases, learn from and reuse the code. Just as we reuse open-source code developed outside the government to save money for the US taxpayer, so too do we offer our code to be reused by our teammates, other agencies, other governmental bodies, or citizens and businesses.

The purpose of this guide is to provide some advice on that reuse and sharing, in hopes of fostering it.

Basic Reuse

All our our code is published and released at GitHub.com under the organization 18F. There you can see all our public repositories (or “repos”). Using only a browser, you can look at any of the code in these repositories, or simply read about the projects. If you wish, you can “follow” a project from beginning to end. Since source code revision systems let you look back in time, you can see the complete history of changes leading up to the current state. Imagine being able to see every draft and edit of Shakespeare’s plays leading up to the publication of the First Folio.

One of the projects we are most proud of and which is highly reusable is FBOpen. FBOpen is a set of open-source tools to help small businesses search for opportunities to work with the U.S. government. FBOpen presents an Application Programming Interface (API) to published Federal contracting opportunities, as well as implementing a beautiful graphical user interface to the same opportunities.

Anyone who wishes to may reuse this code to create their own website, free of charge and unencumbered by obligations. For example, a State could promote economic development within its borders by taking this code, making a slight modification to limit searches to their own State (you would have to be a software engineer to do this, but it is very easy) and then host “Federal Business Opportunities for the Lone Star State.” A business could also build a website using the FBOpen software and API, perhaps even making money by selling advertisements related to the content. The basic idea is that since the software is open source, anyone can use it to build a tool that suits their needs.

Sharing Enhancements

Let us imagine that a business has installed FBOpen, changed the name and branding, and is making some money from ad revenue by target marketing to businesses interested in a particular kind of Federal contract, for example cement and concrete masonry. They’ve made some “masonry specific” changes to the code. In doing so, they realize that they have made the code more modular in some way, an improvement that can be shared back to the Federal Government.

Why would they take the time to share this back to the government, when they won’t get paid for it, and it costs them a small amount of time to do so? Beyond altruism, by doing so they keep their codebase as similar to the official Federal codebase as possible. In this way, when improvements to FBOpen are made by 18F, their software engineers can accept these changes with minimal effort. They may decide that they want to stay up-to-date with the FBOpen codebase, and manage only masonry-specific code extensions.

The mechanism for sharing this code back has been worked out and it is relatively simple, as software engineering goes. It is called a pull request, because it is a request or suggestion to the owner of the codebase to accept or “pull” the code change. It is a formal mechanism which makes crystal clear how the code is changing, which is of course critical. 18F will perform strict code review of all such pull requests, and may simply not accept them at all—not every idea is aligned with the codebase owner’s intentions. In general, however, we welcome such pull requests and enhancements. Just as we hope to create opportunities for American business, we can benefit from the creative output of the entrepreneurs and non-commercial software developers. The taxpayers deserve the least expensive, highest-quality software that we can deliver for their tax dollars.

Some Legal Issues

18F is committed to making our code permissively reusable wherever possible. Work performed by Federal employees, such as the staff of 18F, is not subject to copyright and is in the public domain within the US. However, we use a copyright waiver for other jurisdictions to clarify matters and ensure unrestricted public use outside of the US.

Even though it is our intention to release all code permissively, you may find a derived work of someone else’s code in our repositories. In order to save the taxpayer money, we reuse work that others have created when possible. An example of such a file is pycas.py which is part of the PriceHistory project which was begun by Presidential Innovation Fellows and is now maintained by 18F. This individual file is Copyright Jon Rifkin, 2011, and it was reused and modified as allowed by the Apache License 2.0 under which Mr. Rifkin released it. This file remains copyrighted by Jon Rifkin and covered by the Apache License 2.0.

Since similar situations may arise in any repository, check the individual README and LICENSE files for each project on GitHub for details specific to that project in order to reuse our code legally—which we strongly encourage!

Making Contributions

GSA is not permitted to accept voluntary services or ask people to perform work on open-source projects free of charge. However, since our open-source software projects are available in public repositories for anyone to learn from or reuse, individuals may decide to improve the software for the benefit of others or offer suggestions to improve the code.

In general, individuals who choose to contribute to an open-source project do so without the expectation of payment. There are a variety of reasons why software developers elect to contribute to any open-source software project. The reasons include:

  • Desire to make an improvement to software that a programmer is using;
  • Demonstrating one’s commitment, talent, and experience;
  • Altruism.

In the case of our repositories, there are several kinds of contributions:

  • Report bugs, ideas, requests for features by creating “Issues” at GitHub in our project repositories. An issue that begins “Have you thought of…” could save a project months of labor.
  • Fork our code and play with it, whether you later choose to make a pull request or not.
  • Create pull requests of changes that you think are laudatory. From typos to major design flaws, you will find a target-rich environment for improvements.

Working In Public From Day 1

By Eric Mill

In the wide world of software, maybe you’ve heard someone say this, or maybe you’ve said it yourself: "I’ll open source it after I clean up the code; it’s a mess right now."

Or: "I think there are some passwords in there; I’ll get around to cleaning it out at some point."

Or simply: "No way, it’s just too embarrassing."

These feelings are totally natural, but keep a lot of good work closed that could easily have been open. The trick to avoiding this is simple: open source your code from day 1. Don’t wait for a milestone, don’t wait for it to be stable — do it from the first commit.

Here are a few reasons why you should feel good about working in the open from the moment your shovel goes in the ground:

No one’s going to read your code. Your code is almost certainly boring. Most code is. Instead, people will evaluate your work based on how they’d interact with it. Is it easy to learn how to use it from the README? Is development active? Have many GitHub users starred it? And none of that will matter until your project is far enough along that it’s useful. You will not be in the spotlight until you deserve to be.

You will make better decisions. At the most basic level, you will be vastly less likely to accidentally commit a password to an open source project than a closed one. But more than that: even though no one is reading your code, you’ll still feel a bit of natural pressure to make better decisions. You’ll hardcode less, and move more into configuration files. You’ll make things slightly more modular. You’ll comment more. You’ll catch security holes more quickly. That’s a healthy pressure.

It will not waste your time. It may feel like some of those “better decisions” take more time. But even if you’re the only person who will ever work on this project, you have to live there. You’ll greatly and immediately appreciate having made those decisions the minute you return to your own code after taking a month off. And when making better decisions becomes routine, they stop taking more time — and you become a better coder.

You might just help people. And people might just help you! The internet is a big place and a small world, and GitHub has a way of making unexpected connections. If your work is even a little bit useful to someone else, there’s a good chance they’ll find their way to your door, start poking around, and find a way to be useful right back. Even if you’re working on what you think is the most niche project that no one else would ever use: leave the door open for providence.

Once you get used to beginning your work in public, it stops feeling like performance art and starts feeling like breathing. It’s a healthy routine that produces better work and personal growth, and opens the door to spontaneous contribution and engagement. When your default is open, everyone wins.

18F: An Open Source Team

By Raphael Majma and Eric Mill

At 18F, we place a premium on developing digital tools and services in the open. This means contributing our source code back to the community, actively repurposing our code across projects, and contributing back to the open source tools we use. For a variety of reasons, we believe that doing so improves the final product we create. It is because of this that our policy is to:

  1. Use Free and Open Source Software (FOSS) in our projects and to contribute back to the open source community;
  2. Create an environment where any project can be developed in the open; and
  3. Publish all source code created or modified by 18F publicly.

FOSS is software that does not charge users a purchase or licensing fee for modifying or redistributing the source code. There are many benefits to using FOSS, including allowing for product customization and better interoperability between products. Citizen and consumer needs can change rapidly. FOSS allows us to modify software iteratively and to quickly change or experiment as needed.

Similarly, openly publishing our code creates cost-savings for the American people by producing a more secure, reusable product. Code that is available online for the public to inspect is open to a more rigorous review process that can assist in identifying flaws in the source code. Developing in the open, when appropriate, opens the project up to that review process earlier and allows for discussions to guide the direction of a products development. This creates a distinct advantage over proprietary software that undergoes a less diverse review and provides 18F with an opportunity to engage our stakeholders in ways that strengthen our work.

The use of open source software is not new in the Federal Government. Agencies have been using open source software for many years to great effect. What fewer agencies do is publish developed source code or develop in the open. When the Food and Drug Administration built out openFDA, an API that lets you query adverse drug events, they did so in the open. Because the source code was being published online to the public, a volunteer was able to review the code and find an issue. The volunteer not only identified the issue, but provided a solution to the team that was accepted as a part of the final product. Our policy hopes to recreate these kinds of public interactions and we look forward to other offices within the Federal Government joining us in working on FOSS projects.

In the next few days, we’re excited to publish a contributor’s guide about reuse and sharing of our code and some advice on working in the open from day one.

Take a gander at our /Developer page!

By Gray Brooks

A growing trend both inside government and outside is to have a simple welcoming page for outside developers who may be interested in your team’s efforts. This material is often located at website.gov/developer 1 and points visitors to technical material that developers may be interested in, especially APIs. Collecting technical documentation in one place facilitates the developer experience, ensuring that they can find and begin using APIs with as little friction as possible.

Believing in this best practice, we’re launching 18F’s /developer page to provide a consistent and friendly starting point for other coders — both in the public and within government. There are two main sections, one for our APIs and another for 18F initiatives that are targeted to this community. There’s a good bit there but we hope you’ll check back often as we add more APIs…a lot more APIs!

1: as recommended in the Digital Government Strategy

Midas: A Marketplace for Innovation in Government

By Joe Polastre, 18F and Matt Chessen, Department of State

Midas is an online platform that brings to life the vision of an “Innovation Toolkit” for government. It is a marketplace of skill building opportunities which matches people to projects that they’re passionate about. You can think of it as “kickstarter for people’s time”.

The concept of an Innovation Toolkit started as a Presidential Innovation Fellows project to give federal employees a place to work on innovative ideas across government that may occur outside their normal job or management chain. The Department of State’s Office of eDiplomacy teamed up with The White House Office of Science and Technology Policy to prototype a crowdsourced marketplace for innovation.

The History of Midas

To understand why the State Department was interested in such a project, first imagine that you’re an enthusiastic federal employee that has lots of ideas for government to better serve its people and businesses. One such group of feds is foreign service officers, especially junior foreign service officers who spend their first few years performing consular work. They are located around the world and typically process visas and passports full time. Foreign service officers are highly educated and trained individuals — often with advanced degrees and significant work experience. The question was posed: how can we leverage this untapped potential of the federal workforce to make meaningful progress on innovative projects? And how do we provide professional development opportunities to both local and remote workers that want to build their skills and advance in their career?

From June through December 2013, through the Presidential Innovation Fellows program, we conducted numerous user interviews and meetings with stakeholders at the State Department. We met with foreign service officers, supervisors, and country desk officers that coordinate activities in each region. From these interviews, tests, and prototypes came Midas — a marketplace for projects and opportunities. At State, they started calling it “crowdworking.” (The internal website at State is officially called “Crowdwork”.)

How Midas Works

Midas Project Card

Midas consists of three main components — Projects, Opportunities, and Profiles — all of which are connected through skills and topics of interest.

When employees sign up, they get a profile on Midas (which can be imported from existing services such as LinkedIn). Profiles contain a list of skills as well as topics that employees are interested in or passionate about.

Projects are a container that holds the discussions, events, files, and participants of a proposed activity. Projects also have skills and topics associated with them. This is where work can be coordinated. A project doesn’t have to be a project in the traditional sense; it can be a "tiger team", a community of practice, or an interest group (for example, a group of people that get together to volunteer for a particular cause outside of work).

Midas Opportunity Card

Opportunities are crowdsourced tasks that need volunteers. Opportunities are also associated with skills and topics. An opportunity can be part of a project, or a standalone need. These can be accomplished in as little as 15 minutes, or could be as consuming as a 6 or 12 month full time detail of the employee to another office.

Midas in Action

Midas is currently being piloted by the Department of State [GitHub] and the Department of Health and Human Services [GitHub]. Midas is also a key component of the Office of Personnel Management’s GovConnect initiative, which is part of the President’s Second Term Management Agenda. Through GovConnect, agencies can experiment, prototype, and implement new workforce engagement programs, like Midas.

Crowdwork, the name of the Midas system at the Department of State, received the NextGen Exemplary Group Award. The award will be celebrated at the NextGen Training Summit on July 23, 2014 in Washington, D.C.

Midas for You

At 18F, we are now offering Midas to federal agencies as a Software as a Service. Any agency can test out Midas quickly, and focus on the programmatic aspect of “crowdworking” in their community rather than get bogged down in the details of acquiring and standing up the technology.

Strategic Benefits of using Midas: - Leverage capacity of remote offices to accomplish more within an agency - Augment existing teams (both within and outside agency) with skillsets needed but not currently available within the team - Provide additional professional development opportunities to employees that match their interests and passions - Improve employee morale and engagement

From Day 1, Midas has been available as open source on GitHub. It is free for anyone to download, install, and use. All of our issues, features, and bugs are out in the open. Management of Midas uses agile methodologies and development is organized into sprints.

Come check out Midas for yourself on GitHub, and see how federal employees are working more organically, across traditional silos, and more collaboratively to improve government.

Hot off the press: 18F’s API Standards

By Alan DeLevie and Eric Mill

We recently released the first version of our API Standards — a set of recommendations and guidelines for API production. It is our intention that every 18F API meet these standards, to help us ensure a baseline quality and consistency across all APIs we offer now and in the future.

These standards guide the user-facing implementation details of an API. Wherever possible, the standards prescribe a goal instead of a specific technology. What was once universally recommended about APIs just a few years ago may be dated today. For example, while the best issue tracker may change every couple years, the need for an obvious point of contact to the API producer is more enduring.

The standards are generally technology-neutral, with a few specific opinions when warranted. For example, our standards don’t allow for the use of JSONP, as it raises security and performance concerns.

Though these standards are part of a living document, a focus on goals—and not on tools—will increase their shelf-life. By writing these standards under a goal-oriented and “sane defaults” approach, we hope they help us achieve API objectives that will never change: utility for the users and respect for their time and effort.

We began drafting our standards after forking the White House’s own API standards. By publishing their standards in the open so others could benefit, the White House set an important example—one that we greatly support! Similarly, we invite you to fork our API standards and modify as needed for your own organization’s use.