Professor, Department of Cornell University
Building Smart Memories and Cloud Services with Derecho
The slides for the talk are available here.
The Derecho platform was created to support a new generation of Internet-of-Things (IoT) applications with online machine-learning components. At cloud-scale, such applications require us to build smart memory systems. I’m using this term to refer to a customizable service designed to accept high-bandwidth data pipelines from sources, able to apply machine-learning tools to analyze and understand received content, and offering ways to query the resulting knowledge base with minimal delay. Such services would also need to scale out, yet must maintain their rapid responsiveness and strong consistency.
Derecho, which is now fully implemented (github.org/Derecho-Project), leverages persistent memory and RDMA to solve this problem with exceptional performance and scalability. Derecho is also interesting from a theoretical perspective. In particular, the core protocols used implement Paxos state machine replication in a novel manner optimized for RDMA settings. These protocols have been proved correct, and are also optimal in terms of delay before message delivery, progress during failures and even the mapping to RDMA hardware.
Ken Birman is the N. Rama Rao Professor of Computer Science at Cornell. An ACM Fellow and the winner of the IEEE Tsutomu Kanai Award, Ken has written 3 textbooks and published more than 150 papers in prestigious journals and conferences. Software he developed operated the New York Stock Exchange for more than a decade without trading disruptions, and plays central roles in the French Air Traffic Control System and the US Navy AEGIS warship. Other technologies from his group found their way into IBM’s Websphere product, Amazon’s EC2 and S3 systems, Microsoft’s cluster management solutions, and the US Northeast bulk power grid. His Vsync system (vsync.codeplex.com) has become a widely used teaching tool for students learning to create secure, strongly consistent and scalable cloud computing solutions. Derecho is intended for demanding settings such as the smart power grid, smart highways and homes, and scalable vision systems.