The Contributor’s Guide to 18F: Code for the Common Good
By Robert L. Read, PhD
Transparency in coding makes code more secure. Open-source development is development in the light, sometimes a harsh light, that shows every blemish. At 18F we strongly believe this improves the rapidity of our coding and the quality and security of the code.
We keep the code open to each other, which allows us to quickly scrub in on projects and to dexterously apply the most talented resources to a problem without too much concern for who is formally working on or in charge of a given project. The code is also open to Federal employees from outside 18F and from other agencies. This means that they may both review the code, offers suggestions, and, in some cases, learn from and reuse the code. Just as we reuse open-source code developed outside the government to save money for the US taxpayer, so too do we offer our code to be reused by our teammates, other agencies, other governmental bodies, or citizens and businesses.
The purpose of this guide is to provide some advice on that reuse and sharing, in hopes of fostering it.
All our our code is published and released at GitHub.com under the organization 18F. There you can see all our public repositories (or “repos”). Using only a browser, you can look at any of the code in these repositories, or simply read about the projects. If you wish, you can “follow” a project from beginning to end. Since source code revision systems let you look back in time, you can see the complete history of changes leading up to the current state. Imagine being able to see every draft and edit of Shakespeare’s plays leading up to the publication of the First Folio.
One of the projects we are most proud of and which is highly reusable is FBOpen. FBOpen is a set of open-source tools to help small businesses search for opportunities to work with the U.S. government. FBOpen presents an Application Programming Interface (API) to published Federal contracting opportunities, as well as implementing a beautiful graphical user interface to the same opportunities.
Anyone who wishes to may reuse this code to create their own website, free of charge and unencumbered by obligations. For example, a State could promote economic development within its borders by taking this code, making a slight modification to limit searches to their own State (you would have to be a software engineer to do this, but it is very easy) and then host “Federal Business Opportunities for the Lone Star State.” A business could also build a website using the FBOpen software and API, perhaps even making money by selling advertisements related to the content. The basic idea is that since the software is open source, anyone can use it to build a tool that suits their needs.
Let us imagine that a business has installed FBOpen, changed the name and branding, and is making some money from ad revenue by target marketing to businesses interested in a particular kind of Federal contract, for example cement and concrete masonry. They’ve made some “masonry specific” changes to the code. In doing so, they realize that they have made the code more modular in some way, an improvement that can be shared back to the Federal Government.
Why would they take the time to share this back to the government, when they won’t get paid for it, and it costs them a small amount of time to do so? Beyond altruism, by doing so they keep their codebase as similar to the official Federal codebase as possible. In this way, when improvements to FBOpen are made by 18F, their software engineers can accept these changes with minimal effort. They may decide that they want to stay up-to-date with the FBOpen codebase, and manage only masonry-specific code extensions.
The mechanism for sharing this code back has been worked out and it is relatively simple, as software engineering goes. It is called a pull request, because it is a request or suggestion to the owner of the codebase to accept or “pull” the code change. It is a formal mechanism which makes crystal clear how the code is changing, which is of course critical. 18F will perform strict code review of all such pull requests, and may simply not accept them at all—not every idea is aligned with the codebase owner’s intentions. In general, however, we welcome such pull requests and enhancements. Just as we hope to create opportunities for American business, we can benefit from the creative output of the entrepreneurs and non-commercial software developers. The taxpayers deserve the least expensive, highest-quality software that we can deliver for their tax dollars.
Some Legal Issues
18F is committed to making our code permissively reusable wherever possible. Work performed by Federal employees, such as the staff of 18F, is not subject to copyright and is in the public domain within the US. However, we use a copyright waiver for other jurisdictions to clarify matters and ensure unrestricted public use outside of the US.
Even though it is our intention to release all code permissively, you may find a derived work of someone else’s code in our repositories. In order to save the taxpayer money, we reuse work that others have created when possible. An example of such a file is pycas.py which is part of the PriceHistory project which was begun by Presidential Innovation Fellows and is now maintained by 18F. This individual file is Copyright Jon Rifkin, 2011, and it was reused and modified as allowed by the Apache License 2.0 under which Mr. Rifkin released it. This file remains copyrighted by Jon Rifkin and covered by the Apache License 2.0.
Since similar situations may arise in any repository, check the individual README and LICENSE files for each project on GitHub for details specific to that project in order to reuse our code legally—which we strongly encourage!
GSA is not permitted to accept voluntary services or ask people to perform work on open-source projects free of charge. However, since our open-source software projects are available in public repositories for anyone to learn from or reuse, individuals may decide to improve the software for the benefit of others or offer suggestions to improve the code.
In general, individuals who choose to contribute to an open-source project do so without the expectation of payment. There are a variety of reasons why software developers elect to contribute to any open-source software project. The reasons include:
- Desire to make an improvement to software that a programmer is using;
- Demonstrating one’s commitment, talent, and experience;
In the case of our repositories, there are several kinds of contributions:
- Report bugs, ideas, requests for features by creating “Issues” at GitHub in our project repositories. An issue that begins “Have you thought of…” could save a project months of labor.
- Fork our code and play with it, whether you later choose to make a pull request or not.
- Create pull requests of changes that you think are laudatory. From typos to major design flaws, you will find a target-rich environment for improvements.
Working In Public From Day 1
By Eric Mill
In the wide world of software, maybe you’ve heard someone say this, or maybe you’ve said it yourself: "I’ll open source it after I clean up the code; it’s a mess right now."
Or: "I think there are some passwords in there; I’ll get around to cleaning it out at some point."
Or simply: "No way, it’s just too embarrassing."
These feelings are totally natural, but keep a lot of good work closed that could easily have been open. The trick to avoiding this is simple: open source your code from day 1. Don’t wait for a milestone, don’t wait for it to be stable — do it from the first commit.
Here are a few reasons why you should feel good about working in the open from the moment your shovel goes in the ground:
No one’s going to read your code. Your code is almost certainly boring. Most code is. Instead, people will evaluate your work based on how they’d interact with it. Is it easy to learn how to use it from the README? Is development active? Have many GitHub users starred it? And none of that will matter until your project is far enough along that it’s useful. You will not be in the spotlight until you deserve to be.
You will make better decisions. At the most basic level, you will be vastly less likely to accidentally commit a password to an open source project than a closed one. But more than that: even though no one is reading your code, you’ll still feel a bit of natural pressure to make better decisions. You’ll hardcode less, and move more into configuration files. You’ll make things slightly more modular. You’ll comment more. You’ll catch security holes more quickly. That’s a healthy pressure.
It will not waste your time. It may feel like some of those “better decisions” take more time. But even if you’re the only person who will ever work on this project, you have to live there. You’ll greatly and immediately appreciate having made those decisions the minute you return to your own code after taking a month off. And when making better decisions becomes routine, they stop taking more time — and you become a better coder.
You might just help people. And people might just help you! The internet is a big place and a small world, and GitHub has a way of making unexpected connections. If your work is even a little bit useful to someone else, there’s a good chance they’ll find their way to your door, start poking around, and find a way to be useful right back. Even if you’re working on what you think is the most niche project that no one else would ever use: leave the door open for providence.
Once you get used to beginning your work in public, it stops feeling like performance art and starts feeling like breathing. It’s a healthy routine that produces better work and personal growth, and opens the door to spontaneous contribution and engagement. When your default is open, everyone wins.
18F: An Open Source Team
By Raphael Majma and Eric Mill
At 18F, we place a premium on developing digital tools and services in the open. This means contributing our source code back to the community, actively repurposing our code across projects, and contributing back to the open source tools we use. For a variety of reasons, we believe that doing so improves the final product we create. It is because of this that our policy is to:
- Use Free and Open Source Software (FOSS) in our projects and to contribute back to the open source community;
- Create an environment where any project can be developed in the open; and
- Publish all source code created or modified by 18F publicly.
FOSS is software that does not charge users a purchase or licensing fee for modifying or redistributing the source code. There are many benefits to using FOSS, including allowing for product customization and better interoperability between products. Citizen and consumer needs can change rapidly. FOSS allows us to modify software iteratively and to quickly change or experiment as needed.
Similarly, openly publishing our code creates cost-savings for the American people by producing a more secure, reusable product. Code that is available online for the public to inspect is open to a more rigorous review process that can assist in identifying flaws in the source code. Developing in the open, when appropriate, opens the project up to that review process earlier and allows for discussions to guide the direction of a products development. This creates a distinct advantage over proprietary software that undergoes a less diverse review and provides 18F with an opportunity to engage our stakeholders in ways that strengthen our work.
The use of open source software is not new in the Federal Government. Agencies have been using open source software for many years to great effect. What fewer agencies do is publish developed source code or develop in the open. When the Food and Drug Administration built out openFDA, an API that lets you query adverse drug events, they did so in the open. Because the source code was being published online to the public, a volunteer was able to review the code and find an issue. The volunteer not only identified the issue, but provided a solution to the team that was accepted as a part of the final product. Our policy hopes to recreate these kinds of public interactions and we look forward to other offices within the Federal Government joining us in working on FOSS projects.
In the next few days, we’re excited to publish a contributor’s guide about reuse and sharing of our code and some advice on working in the open from day one.
Take a gander at our /Developer page!
By Gray Brooks
A growing trend both inside government and outside is to have a simple welcoming page for outside developers who may be interested in your team’s efforts. This material is often located at website.gov/developer 1 and points visitors to technical material that developers may be interested in, especially APIs. Collecting technical documentation in one place facilitates the developer experience, ensuring that they can find and begin using APIs with as little friction as possible.
Believing in this best practice, we’re launching 18F’s /developer page to provide a consistent and friendly starting point for other coders — both in the public and within government. There are two main sections, one for our APIs and another for 18F initiatives that are targeted to this community. There’s a good bit there but we hope you’ll check back often as we add more APIs…a lot more APIs!
as recommended in the Digital Government Strategy
Midas: A Marketplace for Innovation in Government
By Joe Polastre, 18F and Matt Chessen, Department of State
Midas is an online platform that brings to life the vision of an “Innovation Toolkit” for government. It is a marketplace of skill building opportunities which matches people to projects that they’re passionate about. You can think of it as “kickstarter for people’s time”.
The concept of an Innovation Toolkit started as a Presidential Innovation Fellows project to give federal employees a place to work on innovative ideas across government that may occur outside their normal job or management chain. The Department of State’s Office of eDiplomacy teamed up with The White House Office of Science and Technology Policy to prototype a crowdsourced marketplace for innovation.
The History of Midas
To understand why the State Department was interested in such a project, first imagine that you’re an enthusiastic federal employee that has lots of ideas for government to better serve its people and businesses. One such group of feds is foreign service officers, especially junior foreign service officers who spend their first few years performing consular work. They are located around the world and typically process visas and passports full time. Foreign service officers are highly educated and trained individuals — often with advanced degrees and significant work experience. The question was posed: how can we leverage this untapped potential of the federal workforce to make meaningful progress on innovative projects? And how do we provide professional development opportunities to both local and remote workers that want to build their skills and advance in their career?
From June through December 2013, through the Presidential Innovation Fellows program, we conducted numerous user interviews and meetings with stakeholders at the State Department. We met with foreign service officers, supervisors, and country desk officers that coordinate activities in each region. From these interviews, tests, and prototypes came Midas — a marketplace for projects and opportunities. At State, they started calling it “crowdworking.” (The internal website at State is officially called “Crowdwork”.)
How Midas Works
Midas consists of three main components — Projects, Opportunities, and Profiles — all of which are connected through skills and topics of interest.
When employees sign up, they get a profile on Midas (which can be imported from existing services such as LinkedIn). Profiles contain a list of skills as well as topics that employees are interested in or passionate about.
Projects are a container that holds the discussions, events, files, and participants of a proposed activity. Projects also have skills and topics associated with them. This is where work can be coordinated. A project doesn’t have to be a project in the traditional sense; it can be a "tiger team", a community of practice, or an interest group (for example, a group of people that get together to volunteer for a particular cause outside of work).
Opportunities are crowdsourced tasks that need volunteers. Opportunities are also associated with skills and topics. An opportunity can be part of a project, or a standalone need. These can be accomplished in as little as 15 minutes, or could be as consuming as a 6 or 12 month full time detail of the employee to another office.
Midas in Action
Midas is currently being piloted by the Department of State [GitHub] and the Department of Health and Human Services [GitHub]. Midas is also a key component of the Office of Personnel Management’s GovConnect initiative, which is part of the President’s Second Term Management Agenda. Through GovConnect, agencies can experiment, prototype, and implement new workforce engagement programs, like Midas.
Crowdwork, the name of the Midas system at the Department of State, received the NextGen Exemplary Group Award. The award will be celebrated at the NextGen Training Summit on July 23, 2014 in Washington, D.C.
Midas for You
At 18F, we are now offering Midas to federal agencies as a Software as a Service. Any agency can test out Midas quickly, and focus on the programmatic aspect of “crowdworking” in their community rather than get bogged down in the details of acquiring and standing up the technology.
Strategic Benefits of using Midas:
- Leverage capacity of remote offices to accomplish more within an agency
- Augment existing teams (both within and outside agency) with skillsets needed but not currently available within the team
- Provide additional professional development opportunities to employees that match their interests and passions
- Improve employee morale and engagement
From Day 1, Midas has been available as open source on GitHub. It is free for anyone to download, install, and use. All of our issues, features, and bugs are out in the open. Management of Midas uses agile methodologies and development is organized into sprints.
Come check out Midas for yourself on GitHub, and see how federal employees are working more organically, across traditional silos, and more collaboratively to improve government.
Hot off the press: 18F’s API Standards
By Alan DeLevie and Eric Mill
We recently released the first version of our API Standards — a set of recommendations and guidelines for API production. It is our intention that every 18F API meet these standards, to help us ensure a baseline quality and consistency across all APIs we offer now and in the future.
These standards guide the user-facing implementation details of an API. Wherever possible, the standards prescribe a goal instead of a specific technology. What was once universally recommended about APIs just a few years ago may be dated today. For example, while the best issue tracker may change every couple years, the need for an obvious point of contact to the API producer is more enduring.
The standards are generally technology-neutral, with a few specific opinions when warranted. For example, our standards don’t allow for the use of JSONP, as it raises security and performance concerns.
Though these standards are part of a living document, a focus on goals—and not on tools—will increase their shelf-life. By writing these standards under a goal-oriented and “sane defaults” approach, we hope they help us achieve API objectives that will never change: utility for the users and respect for their time and effort.
We began drafting our standards after forking the White House’s own API standards. By publishing their standards in the open so others could benefit, the White House set an important example—one that we greatly support! Similarly, we invite you to fork our API standards and modify as needed for your own organization’s use.
Intro to APIs: Working with URLs, JSON, APIs, and Open Data — Without Writing Any Code
June 27 @ 9:30am - 11:30am
GSA’s digital teams are offering a user-friendly intro course to APIs. Regardless of your skill level, you will walk away from this lesson understanding what APIs are and how developers use them.
Every federal agency is scaling out its use of APIs so even if they are not a part of your normal daily work, learning what they are, how they work, and why they are changing government.
This is an in-person, interactive session led by Eric Mill, a key developer on GSA’s 18F team and an expert in open data and government technology. Again, this session is designed so that even complete beginners will be able to follow along, so if anyone you know is wondering what all the fuss about APIs is about – this is for you.
Who should attend:
- System Owners
- Digital Engagement Staff
- CIO and CTO Staff
- Government Staff Interested in Open Data efforts
- Anyone Who Is Curious!
What you’ll need:
- A laptop, with a working connection to the public Internet
- A recent version of Firefox or Chrome
- An extension installed for Firefox or Chrome to view JSON in your browser (please install beforehand)
Links that will come up:
Friday, June 27, 2014
9:30 am - 11:30 am
GSA Central Office
1800 F St NW, Washington, DC 20006
Announcing The /Developer Program: A New Hub for Federal API Creators
By Leah Bannon and Gray Brooks
We recently launched our /Developer Program (pronounced “slash developer”) to help federal agencies develop useful, robust APIs. The Program is a collection of educational resources, opportunities to engage the community for help and feedback, and tools that can help you build APIs — essentially an ever-growing knowledge base curated by 18F. We affectionately named the /Developer Program after the common practice of providing API documentation and links on government websites at website.gov/developer1.
We use the term “launched” liberally though, because this has been a labor of love over the past year or more for the digital gov community. APIs are basically a way for two programs to share information or perform an action2, and they continue to play an increasingly fundamental role across the web. There are now almost 100 developer hubs that provide access to hundreds of web services across government.
Whether you’re new to APIs or have years of experience, the site’s list of educational resources can help you better understand the role that APIs can play for your agency as well as explain them to your colleagues. These resources include API definitions and analogies, training, best practices, talking points, FAQs, and case studies in case you need inspiration or examples.
Community & Collaboration
The best way to improve and promote your agency’s APIs is to get involved with API producers at other agencies and the developers who use your APIs. The site offers links to meetups and other API events, as well as our API Usability Testing program where agencies demo their APIs to real users and customers to get their feedback. We also provide examples of excellent government APIs, sample policy and contract language for building APIs, and links to good API standards and guidelines.
Tools & Resources
We are compiling a list of open source tools and resources to help you actually build your APIs, including data converters, standards, developer kits, and more. There are hosted tools that can take data in a spreadsheet file and instantly make it available as a RESTful API. For enterprise needs, a powerful API proxy offers key management and analytics functionality for any agency’s use.
Please feel free to provide feedback and suggest additional resources; the entire /Developer Program is an open source project on GitHub.
This was recommended in the Digital Government Strategy.
Packaging Up API Usability Testing for Agency Re-Use
By: Gray Brooks
Over the past year, a GSA collaboration has seen a project that offers API Usability Testing to federal agencies go from the pilot stage to a regular, robust series. Already, 13 agencies and programs have participated, and several more participate with every monthly session that passes. The best examples from across the government have made clear that one of the most important tasks of API producers is to regularly engage their developer community and listen to what they have to say. But just encouraging agencies to do this only goes so far.
In addition to offering any federal agency direct support by hosting them at an API Usability session, we’ve also been working to document and open-source the actual process itself. We want every agency to realize how readily they can scale out these exercises on their own. Within the API Usability Kit, agencies will find proposed schedules, email and handout templates, an overall guide, and frequently asked questions. But since each component is itself an open source project, anyone can easily edit any page or fork the entire project.
As we move forward, it’s our hope that you’ll join us — in person for API Usability sessions for your program, but also through open collaboration with the material that we use to run the program. Help make this program better! Anytime you have a suggestion, all you have to do is click the ‘Edit’ button in the sidebar and propose away.
As with any good open-source project, we eat our own dog food and strive to ensure that this program grows through an open, organic process. We’ll continue to edit and build out the resources that make up the API Usability Kit. Through that, we believe we can ensure the best product for agencies and the best results for the final customer — the people of the United States.
See also, Gray’s presentation from last week’s 18F Demo Day:
Slides from the Inaugural 18F Demo Day
The presentations given at the inaugural 18F Demo Day on May 9, 2014 are online and available at Speaker Deck. If you would like more information on any topic, please feel free to contact the individual speaker.
Hacking Bureaucracy —
FBOpen & Procurement Tools —
Midas / Innovation Toolkit —
API Usability Testing —
SAM.gov “Pizza Tracker” —
Robert Read & Navin Vembar
CAP Communicart —
We the People: User-Centered Design —