Netflix Moves Its Service to the Cloud
By Bob Violino
Netflix, a Los Gatos, Calif., company that provides movies and television programming to subscribers over the Internet, faced significant challenges in getting its service running in the cloud.
The move to the cloud "required re-architecting our monolithic backend into small individual services, [for example] moving to a service-oriented architecture," says Carl Quinn, software engineer at Netflix, which has more than 30 million members in more than 50 countries. "Since we are a Java shop and like to share code, our approach was to have each service team publish client libraries for communication to their services."
These libraries, plus all the shared code that was refactored into common libraries, created a very large pool of modules, which are constantly being updated and expanded, Quinn says.
To solve its complex application development challenges by streamlining processes, the company deployed a product called Artifactory from JFrog. The technology enables software developers and development teams to manage, host and control the flow of binary artifacts from development to production.
An artifact is something that is produced by a tool, Quinn says. In this case, binary artifacts "are the binaries produced by our build tools—either the shared libraries or the application bundles," he explains.
Artifactory allows Netflix developers to more efficiently access, share and deploy its build artifacts, saving a significant amount of time. Migrating from a "primitive SCM-based library management system" to one centered around Artifactory allowed Netflix to manage its giant library pool with a small team of tool developers in partnership with the module-owning teams, Quinn says.
"This is a complex problem, but the combination of performance and the feature set Artifactory provided made it possible," Quinn says. "From start to finish, we were able to implement this in a very short amount of time. In under six months, we had updated our build tools to use Ivy and infrastructure around Artifactory."
Ivy is a dependency management system that is an open-source Apache project. It "describes the dependency relationships between the binary artifacts in metadata files, and is used in a build to determine which libraries are needed for a given project," Quinn says. "Artifactory supports Ivy metadata files for each module."
It's difficult to quantify the exact savings Netflix has derived from using the JFrog technology, Quinn says. "What we achieved was more of an unshackling of product teams from having to manage the details of the build and code sharing," he adds. "We enabled 10 to 20 teams to develop cloud services and client libraries without having to designate specific team members to build management."
Artifactory has benefitted Netflix' overall business goals in a number of ways. "Our engineering tools team provides the tools and infrastructure services for build, test and deployment for all server-side systems in the [Amazon.com] AWS cloud and legacy data center," Quinn says. "This includes all the systems supporting the Website, movie discovery and streaming playback."
The initial migration to the cloud involved a lot of significant re-architecting and refactoring of code, which would have been essentially impossible without a simple self-service continuous integration pipeline to streamline the development and integration overhead from this work, Quinn says.
"Now that we are in the cloud, our continuous integration system with Artifactory allows us to maintain a large pool of shared code across many teams, and propagate new fixes and features quickly from the core teams out to the various product teams," he says.