How Wal-Mart hired 2,200 developers and made 'magic'
In addition to all the web traffic that a company like Wal-Mart has to deal with, there's all kinds of complex wizardry that goes into wrangling a tremendous amount of data — millions of items, their prices, customer recommendations, warehouse availability — into a website that you can actually search and buy stuff from.
This is why four years ago, Wal-Mart opened its Walmart Labs division in the heart of Silicon Valley's Mountain View, not far from Google, LinkedIn, and Yahoo.
It started from Wal-Mart's acquisition of a 65-person search-and-analytics startup called Kosmix in 2011, but now it employs 2,200 "technologists" — developers, engineers, data experts, and the like — to solve the retailer's biggest digital problems.
"When you build a product for Wal-Mart, millions of people see it," says Wal-Mart SVP and Chief Technology Officer Jeremy King, who heads up Walmart Labs.
Walmart.com sees "millions of transactions" every day, he says. And when you give smart people access to the data and resources they need, you get "magic," King says. It's not a word he uses lightly.
"I'm a big Ayn Rand fan, and magic is a bad thing in that world," King says.
Online and offline were separate worlds
Wal-Mart has had a technology presence in northern California since the year 2000, but at that point, the company's site and its stores were a total mismatch: Not every item from the stores was available online, and even Wal-Mart's internal item numbers weren't consistent between the stores and the website.
It made life difficult for customers and Wal-Mart alike, especially as Amazon's star continues to rise. Around 2011, Wal-Mart CEO Mike Duke knew it was time to take action and attack ecommerce sooner rather than later, or else risk getting overshadowed.
He wanted developers.
It was an aggressive approach — Duke was supportive and wanted to give Walmart Labs whatever it needed to work, but his attitude was something like "don't call home until you get results," King recalls.
It wasn't long before King realized that stuff had to change on the back end, he says. The company was using a mishmash of different technologies, some of it outsourced to outside vendors and some of it not.
They were collecting massive pools of customer data, but it was just sitting there, unused.
"We have to dump these vendors," King recalls thinking early on in his Wal-Mart career.
And that's what they did, opting to build their own systems that could connect developers to data (though a Wal-Mart spokesperson was quick to note that it still engages with vendors on stuff like storage and networking).
King boasts that Wal-Mart is one of the earliest and biggest users of Apache Hadoop, the hot data-crunching technology that businesses big and small are making use of to build better, smarter applications.
Similarly, when it came to building out a cloud to efficiently support all of the traffic coming to the site, Walmart Labs became one of the first big users of OpenStack, a free cloud technology that's still finding its way into the traditional enterprise.
The important part, King says, is that whatever they build, they build it themselves — often contributing the things they've learned back to the developer world as open source. Plus, the promise of working with hot technologies at huge scales is a great recruitment pitch, he says.
"You can't outsource innovation," King says. "It's important to own and build it at that scale."
The advantage is that Wal-Mart's developers are closer to the data and closer to the systems that actually support Wal-Mart's customers every day.
Plus, the data means that they can optimize Wal-Mart's logistics, too, King says. If a lot of people in one area are ordering, say, diapers, Walmart.com's software can detect that and allocate more diapers to that area's warehouses.
"In the end, you need to build a team that has daily access to the life of the customer," King says.
Walmart Labs has no direct place in the brick-and-mortar operation of the company, King says, but he has a "great relationship" with Wal-Mart CIO Karenann Terrell. And as the lines between ecommerce and in-store shopping get blurrier, they're working together closely.
"To make that kind of change, it's not Walmart Labs sitting in the corner," King says.