Five key principles that make geographically split software teams work

Geographically distributed teams can work. How? In my experience, there are five key principles that make all the difference.

The other day, I realized that I have worked on geographically split software teams for the last decade. I didn’t set out intending to do so. When I got my first job out of college as a software engineer, I thought being in the office was non-negotiable. If you ask company recruiters, most say in no uncertain terms they are not looking for remote employees.

But the reality on the ground is that many software companies end up letting most full-time engineers work flexible hours, from anywhere. Yet few companies acknowledge or advertise this explicitly. You can’t get this opportunity by asking for it in your interview, but after you are in the door, the opportunity arises. There are a few ways this happens.

Software companies with heavy on-call burdens, like Amazon for example, end up with an unspoken but widespread assumption that people can work from anywhere because they are working all the time. Since most engineers are contractually obligated to be on-call, and emergency issues arise at all hours of the day and night, these engineers start working remotely out of necessity. Soon, whole teams realize there is no barrier to working where and when they are most productive. On any given day during my tenure at Amazon, maybe half of my teammates weren’t in the office when I was.

 I’ve worked for several startups with surprisingly similar team environments. At one, people preferred to work at nearby coffeehouses with outdoor porches that made it easier to work while smoking. At another, unreliable wireless bandwidth in the office drove people to work from home out of necessity. At another, almost every person employed there lived in a different time zone, because this startup needed a very specific set of skills that happened to be scarce at the time.

Later I went to work for Rosetta Stone, which had three offices when I started, and grew to six offices when I left . Most of the teams I worked on ended up having people in at least three office locations, and as many time zones. Only one of the four bosses I had during those years worked in the same office as me. Not only were the teams geographically split, but once a team is split across two or more time zones, for all practical purposes, this team is also working flex time hours.

 These teams all worked well together. No matter where and when we worked, everyone still was accountable for meetings and deadlines. I never felt particularly disconnected from my team, and I had rapport with and support from my bosses.

Today I work for Pedago.com, a startup company that is geographically split and has been from the very beginning. It is the most productive team I have ever worked with.

Geographically distributed teams can work. How? In my experience, there are five key principles that make all the difference.

1. Shared chat room where all team communication, questions, updates, and banter happen

I’ve used Skype, IRC, and HipChat for this — there are many viable options. In a geographically split team, showing up to the chat room is the same as showing up to work. Conversely, not showing up to chat is like not showing up to the office, with the same professional and social side effects. Knowing everyone will be in one place means you always know where you can ask questions if you have them. And if you are ever away and need to catch up on team discussion, there’s a nice transcript of all the debates, conversations, and decisions the team made while you were away that you can easily read to catch up.

2. Shared “online” hours

These are the hours everyone is guaranteed to be available in the shared chat room. No matter how scattered the team is, there will always be some set of hours that work for everyone. Everyone is assumed to be available during these hours, and if any person can’t make it or needs to leave to run errands, they are expected to notify everyone.

3. Short daily video check-in meetings

If your team uses Scrum as its project management strategy, these are just your daily standup meetings. It makes a huge difference to see peoples’ faces once a day, if only to remember that they are human beings in addition to being demanding product owners or cantankerous engineers. The visual feedback you get from face to face conversations helps facilitate complex discussions, amplifies positive feedback, and the subconscious pressure to announce a socially acceptable level of progress helps hold everyone accountable.

4. Teammates need to be aggressive about helping and asking for help.

However tempting, no one should ever spin their wheels when stuck. Teams should declare that ad hoc pair programming, calls, and hangouts are a part of their team’s working style, and developers should be aggressive about picking up the phone or screen-sharing whenever they get stuck or need to ask a question.

5. Everyone on the team needs to buy into these rules.

No exceptions. If even one person refuses to get in the shared team chat-room or doesn’t feel like showing up to the video check-in meetings every day, these strategies won’t work for the rest of the team. Everyone on the team has to buy in, and in particular, managers and team leads need to lead by example, being disciplined about leading discussions and disseminating information only in the team’s shared forums.

But how do you know people are working if you can’t see them?

Simple answer. The same way you know they’re working when you can see them: you talk to them about requirements, check in on progress, look at what they deliver. If you are a technical manager, you monitor deployments or skim the latest check-ins. Software management really isn’t very different with distributed versus in-house teams. The secret to success is still the same: clear direction, well-defined, prioritized requirements, and carefully managed execution.

Build and Deploy with Grunt, Bamboo, and Elastic Beanstalk

In response to Twitter feedback on our recent post “Goodbye, Sprockets! A Grunt-based Rails Asset Pipeline,” we at Pedago would like to share an overview of our current build and deploy process.

In response to Twitter feedback on our recent post “Goodbye, Sprockets! A Grunt-based Rails Asset Pipeline,” we’d like to share an overview of our current build and deploy process.

It goes a little something like this:

Local development environment

We currently have a single git-managed project containing our Rails server at the top level and our Angular project in a subdirectory of vendor. Bower components are checked in to our repo to speed up builds and deploys. The contents of our gruntfile and the organization of our asset pipeline are described here.

We can start up our server via grunt server (which we have configured to shell out to rails server) or directly with rails server for Ruby debugging.

Even though both the client and server apps are checked into the same project and share an asset pipeline, we restrict our Angular code to only communicate to the backend Rails server over APIs. This enforces a clean separation between client and server.

Bamboo build project

When Angular and Rails code is checked in to master, our Bamboo build process runs. We always push through master to production, à la the GitHub flow process. The build process comprises two stages:

Stage 1: Create Artifacts:

  • Rails: bundle install and freeze gems.
  • Angular: npm install, grunt build. No bower install is needed because we check-in our bower_components. The grunt build step compiles, concatenates, and minifies code and assets. It also takes the unusual step of cache-busting the asset filenames and rewriting any references in view files to point to the new filenames.
  • The resulting artifact is saved in Bamboo and passed to Stage 2.

Stage 2: Run Tests:

  • Rails: run rspec model and controller tests, and then cucumber integration tests. It was a bit tricky to get headless cucumber tests running on Bamboo’s default Amazon AMI; see details in our previous blog post.
  • Angular: grunt test.

If the artifact creation succeeds, and the tests run on that artifact all pass, Bamboo triggers its associated deploy project. Otherwise, our team receives notifications in Hipchat of the failure.

Bamboo deploy project

After every successful build, Bamboo is configured to automatically deploy the latest build to our staging environment.

The Bamboo deployment project runs the following tasks to kick off an Elastic Beanstalk deployment:

  1. Write out an aws_credentials file to the build machine. We don’t store any credentials on our custom AMIs. Instead, we keep them in Bamboo as configuration variables and write them out to the build machine at deploy time.
  2. Run Amazon’s AWSDevTools-RepositorySetup.sh script to add aws.push to the set of available git tasks on the build machine.
  3. Kick off the deployment to our Elastic Beanstalk staging environment with a call to git aws.push from the build machine’s project root directory.

Since our project is configured to use Elastic Beanstalk, the remaining deployment-related configuration (like which Elastic Beanstalk project and stage to push the update to) is checked in to the .elasticbeanstalk and .ebextensions directories in our project and made available to the git aws.push command. If there is interest in sharing the contents of these config files, please let us know on Twitter.

Elastic Beanstalk staging environment

After the staging deployment has been kicked off by Bamboo, we can head over to our EB console at https://console.aws.amazon.com/elasticbeanstalk and monitor the deployment while it completes. The git aws.push command from the previous step is doing the majority of the work behind the scenes. For staging, we use Amazon’s default Rails template, and “Environment type: Single instance.” Amazon’s default Rails template manages Rails processes on each server box with a passenger + nginx proxy.

When we first decided to go to a grunt-based asset pipeline, we worried this might impact the way we deployed our servers. In fact, it does not. Our git code bundle containing our Rails app, Angular front-end, and shared assets is deployed to Elastic Beanstalk via git aws.push, exactly as it was prior to our grunt-based asset pipeline switch.

We then do smoke testing on our staging environment.

Elastic Beanstalk production environment

After we have determined the staging release is ready to go to production, we promote the current code bundle from staging to production simply by loading up the EB console for the production stage of our project, clicking “Upload and Deploy” from the Dashboard, clicking “All Versions” in the popup, then selecting the git version currently deployed to staging.

For production, we use Amazon’s default Rails template, and “Environment type: Load balanced, auto scaling.” Elastic Beanstalk takes care of rolling updates with configured delays, aka no-downtime deployments.

Wrap up

The above system, combined with the grunt-based asset pipeline described in our previous post, allows us to iterate and deploy with confidence. Future work will focus on improving deploy times, perhaps by baking AMIs or exploring splitting our monolithic deployment artifact into multiple pieces, e.g., code and assets, npm packages, etc.


Curious about Pedago online education? Enter your email address to be added to our beta list.

Questions, comments? You should follow Pedago on Twitter.

Headless integration testing using capybara-webkit

We use Cucumber for integration testing our Rails servers, and by default all Cucumber scenarios tagged with “@javascript” pop up a browser. We needed to get this running headless so we could run these tests on our build machine. We use the Atlassian suite, and Bamboo for CI, running on EC2.

This post is for developers or sysadmins setting up Rails integration testing on a CI system like Travis, Hudson, or Bamboo.

We use Cucumber for integration testing our Rails servers, and by default all Cucumber scenarios tagged with “@javascript” pop up a browser. We needed to get this running headless so we could run these tests on our build machine. We use the Atlassian suite, and Bamboo for CI, running on EC2.

The de facto way of running headless tests in Rails is to use capybara-webkit, which is easy to install and run locally following the guides here.

Capybara-webkit relies on Qt, which is straightforward (though slow) to install on OS X, which we use for development. Our build box however is Amazon Linux, which is supposedly a distant cousin of CentOS. We’re using Amazon Linux because Bamboo OnDemand provides a set of stock Amazon Linux AMIs for builds that we have extended and customized.

We started out following the CentOS 6.3 installation guide from the capybara-webkit wiki above but quickly encountered problems because Amazon Linux doesn’t ship with a lot of libraries you might expect from Redhat or CentOS, like gcc and x11.

Here are the steps we followed to get Qt installed and our headless Cucumber tests running on our Bamboo build machine. This installation process was tested on ec2 AMI ami-51792c38 (i686).


# First install dependencies listed on http://qt-project.org/doc/qt-4.8/requirements-x11.html that do not ship with amazon linux amis.
# If you dont do this, ./configure below will fail with errors like "Basic XLib functionality test failed!"

yum install -y gcc-c++
yum install -y libX11-devel
yum install -y fontconfig-devel
yum install -y libXcursor-devel
yum install -y libXext-devel
yum install -y libXfixes
yum install -y libXft-devel
yum install -y libXi-devel
yum install -y libXrandr-devel
yum install -y libXrender-devel
 
# download, configure, and install qt from source
wget http://download.qt-project.org/official_releases/qt/4.8/4.8.5/qt-everywhere-opensource-src-4.8.5.tar.gz
tar xzvf qt-everywhere-opensource-src-4.8.5.tar.gz
cd qt-everywhere-opensource-src-4.8.5
./configure --platform=linux-g++-32
 
# caution: this will take a long time, 5 hrs on an m1.small!
gmake
gmake install
 
# add qmake location to path
export PATH=$PATH:/usr/local/Trolltech/Qt-4.8.5/bin/
 
# now finally gem install will work!
gem install capybara-webkit

Curious about Pedago online education? Enter your email address to be added to our beta list.

Questions, comments? You should follow Pedago on Twitter.

Learning Game Trees and Forgetting Wrong Paths

This is the second of two blog posts delineating the pedagogical approach of Herb Simon, the man credited with inventing the field of artificial intelligence, for which he won a Turing award in 1975. (Read the first post here.) Simon was a polyglot social scientist, computer scientist and economics professor at Carnegie Mellon University, He later won the Nobel Prize in 1978 in economics for his work in organizational decision-making.

Game Tree
Tic Tac Toe Game Tree, Gdr from Wikimedia Commons

Dr Simon would often tell his students that he liked to think about human learning as a game tree: when you start out learning about a new topic, you begin at the root of the tree with what you already know, and follow connections to related topics, discovering new “nodes” in the tree. You employ a variety of search strategies to follow connections both broadly and deeply through related topics, loading as much of the explorable tree into memory as possible. As you discover and master each “node” on the tree, you learn which branches of the tree are fruitful and which are fruitless.

During and after exploration though, the entire game tree remains in your working memory, slowing you down. When you take breaks, not only are you relaxing, but you are also forgetting wrong paths – pruning those fruitless branches from your working memory. When you next return to the task at hand, you resume exploring connections and mastering concepts not at the very top of the tree, but in the most fruitful subtrees where you left off, making better use of your working memory.

At Pedago, we believe in learning by doing, and we want to break complex topics and concepts down into what Seymour Papert in the book Mindstorms calls “mind-sized bites.” One of the benefits of breaking complicated topics into “bites” is that it is easier to build learning content that learners can work through when they only have a few minutes free, on whatever device they have on hand.

As we build our database of short concepts and lessons, we find ourselves also building a rich tree structure of topic relation metadata that in structure is not unlike Simon’s game tree of learning. A nice side-effect of a learning solution with rich, encapsulated, short lessons is that you don’t have to commit to a thirty minute video – you can learn in bits and pieces throughout your day. And by doing this, you are unintentionally building and then pruning your learning game tree in an efficient way, forgetting wrong paths and making the best use of your working memory each time you return to your lessons.

 

Herb Simon on Learning and Satisficing

This is the first of two posts delineating the pedagogical approach of Herb Simon, credited with inventing the field of AI, for which he won a Turing award in 1975.

This is the first of two blog posts delineating the pedagogical approach of Herb Simon, the man credited with inventing the field of artificial intelligence, for which he won a Turing award in 1975. Simon was a polyglot social scientist, computer scientist and economics professor at Carnegie Mellon University. He later won the Nobel Prize in 1978 in economics for his work in organizational decision-making.

Herbert Simon in front of blackboard
Herbert Simon, Pittsburg Post Gazette Archives

“Learning results from what the student does and thinks and only from what the student does and thinks. The teacher can advance learning only by influencing what the student does to learn.” –Herb Simon

Among his many accomplishments, Herb Simon was a pioneer in the field of adaptive production systems. He also identified the decision-making strategy “satisficing,” which describes the goal of finding a solution that is “good enough” and which meets an acceptability threshold, as opposed to “optimizing,” which aims to find an ideal solution.

Simon believed that human beings lack the cognitive resources to optimize, and are usually operating under imperfect information or inaccurate probabilities of outcomes. In both computer algorithm optimization and human decision-making, satisficing can save significant resources, as the cost of collecting the additional information needed to make the optimal decision can often exceed the total benefit of the current decision.

We live in a world where overwhelming amounts of information are at our very fingertips. Every month new educational software offerings are on the market. You can find tutorials to fix anything in your house, learn a new language for free, find lessons that teach you to dance, and watch video lectures from top universities in the topics of your choice.

I like to think of myself as a polyglot learner: I would love nothing better than to just take a year, or two, or ten, and learn as much as I can about everything. But unfortunately, I have limited time. How do I know which tutorials, lessons, and classes are worth the commitment of my time? How can I find a satisficing solution to the problem of becoming a more well-rounded learner and human being?

In Simon’s words, “information is not the scarce resource; what is scarce is the time for us humans to attend to it.” At Pedago we’ve been inspired by thinkers such as Simon to build a learning solution that makes the most of the scarce resource of your time, by employing curated streams of bite-sized lessons; rich, explorable connections between topics; interactive learn-by-doing experiences; and just the right amount of gamification. We want to enable you to craft your own learning experience, so that you can, as Simon would say, positively influence what you do and what you think.

Stay tuned for the second post in this series as we examine Simon’s modeling of human learning.