Building Upstream: De-Risking Dependencies for Software Development
In distributed software systems, it’s very tempting to envision and even plan the development effort in the direction of dependencies → Downstream. Data flows from an originating source through an API to its final destination: the user-experience. So of course you have to build those things in that order… right?
Well, no. Because of the way modern distributed applications and websites work, it’s actually quite a bit more efficient — and can also be much faster — to design the system in the opposite direction, and then build the pieces concurrently. I call this “Building Upstream.”
Data Streams
First let’s explain our metaphor. Imagine an app or a website, and all the systems that make it work, as a stream. A stream has a top, and a bottom, and the water flows downhill. But instead of water, the stream is carrying data. Data starts at the top, at the source, and flows all the way down to the bottom, where the user is waiting.
Building Upstream
So, let’s start at the beginning… which is to say, the end. Instead of building the data source first, imagine starting at the other side of the system — the UX. If we design the UX we want first, we can step backwards up the stream and back out the data sources we’ll need at the end. Which is to say, the beginning.
Let’s look at an example.
I’m building a Library app, and it will list all the media the library has to offer, so that users can browse and reserve stuff. So let’s build the UI first.
My UI is basically just a sorted grid of books, but I’ll need a few data points to draw it. I can infer by looking at the design that I’ll probably need the url of the image, the title, author and whether the book is checked out. Right there, I’ve already backed out my first step… what data points are necessary to draw the UI. Knowing that allows me to craft a payload that could drive the screen:
Now, I’ve effectively worked backwards from the UI, extracted the data points I’ll need, and designed the payload I want my middleware API to deliver to my front end. So now I go find the data, build the API, and then come back to build my app …right?
Well sure, you could do that. But the beauty of this approach is that all of your front end development is now unblocked. You can create mocks of your payload that express different conditions, and build and test your complete application code — networking, data modeling, presentation logic and layouts — without depending on any deliveries from any other teams. You can even test your app, as long as you re-test against real data later on.
Wait what? So I just build my front end and I’M DONE?
Well not quite. I said you could build your front end… but you can’t release it yet. A distributed software product requires many layers, and we’re only building one. The API and data source still need to be developed. But now ayou don’t have to wait for those other layers to be functional before you can build stuff. Everyone can work independently, and then meet in the middle. And that will be the end.
End to End Integration
The final step in an upstream build is to connect all the parts and see what happens. Remember that, in order to accelerate development, the front-end has been pointing at mocked-up APIs that provide payloads that it expects will be delivered by the finished middleware, based on the production data-source… but it doesn’t always work out that way. Maybe some of the data couldn’t be provided by certain APIs the way you expected, or maybe some data flat-out wasn’t available at all — but there are always adjustments to be made in the integration step.
Keep in mind, though, that the best way to avoid surprises at integration time is to communicate with other teams regularly, and document every decision point and discovery. Just because you’re building independently doesn’t mean you’ve removed the dependencies altogether; all you’ve done is removed the possibility of the work of one team blocking the work of another. In the end, the product needs all of its distributed parts in order to function, so for better or worse, you’re in it together.
Building upstream is the best way to alleviate dependency blockers and accelerate development when all the parts are built concurrently. Just don’t lose track of other teams progress until the very end, or you might be in for a big surprise.
