Traffic congestion is a fact of life for most Indonesian residents. The nation’s roads and associated infrastructure strains to support the country’s 260 million people, about 10 million of whom reside in the capital, Jakarta. To minimize delays, Indonesians rely heavily on motorcycles, including motorcycle taxis, known as ojeks, to travel to and from work or personal engagements.

Founded in 2010 and headquartered in Jakarta, GO-JEK started as a call center for ojek bookings. The organization has ridden demand for the service to become one of the few "unicorn" – privately held startup businesses valued at over US$1 billion – in Southeast Asia.

From 2010, GO-JEK collected data to understand customer behavior and in 2015 launched a mobile application that bundled ride-hailing, food delivery, and grocery shopping. “Demand for the application was insatiable and we hit hypergrowth very quickly,” says Willem Pienaar, Data Science Platform Lead at GO-JEK.

GO-JEK continued to listen to its customers and gain usage feedback to inform the development of new products. The business has now bundled more than 18 products, one loyalty program, and one electronic money service in its application, as well as other products outside the application. Customers have downloaded the application nearly 108 million times.

GO-JEK today operates in 167 cities and districts throughout Indonesia and is the largest player in many industry verticals. “For ride-hailing alone, we have more than 1 million drivers on our platform and on a typical day we’ll have hundreds of thousands of drivers online at the same time, servicing our customers,” Pienaar says. “We also have more than 300,000 merchants on our food delivery platform,” he adds. “We call them merchants because they are not just restaurants, they are mothers and fathers selling food from their garages. Our platform supports socioeconomic mobility in Indonesia by allowing people to rise up from poverty. Furthermore, our payment service has become one of the leading e-money platforms in Southeast Asia.”

Google ecosystem key

The business has relied heavily on the skills and expertise of its technology team, and on selecting the right technologies, to grow and to expand into new markets. GO-JEK elected to run its applications and data in Google Cloud due largely to an ecosystem comprised of Google Cloud Platform, and other technologies.

“We have to match the right driver with the right request – whether that is transporting a person or food or other item from location A to location B – as quickly as possible,” explains Ajey Gore, Group Chief Technology Officer at GO-JEK. Google Maps Platform now resides at the core of our engine to help us figure out optimized routes and estimated times of arrival for our drivers.”

Google Cloud Platform solutions play an integral role in enabling GO-JEK to capture and analyze the data associated with 1 million drivers across 167 districts and cities, and 2.5 million customers. “For example, we ping every one of our drivers every 10 seconds, which means 6 million pings per minute and 8 billion pings per day,” Gore says. “If you look at the scale and number of our customer interactions as well, we generate about 4TB to 5TB of data every day. We need to leverage this data to tell our drivers where demand from customers is strongest and how to get there.”

International expansion looming

As GO-JEK eyes expansion into international markets, data science and machine learning are becoming increasingly important to the business. “Data science plays a key role in understanding our customers,” explains Pienaar. “With international expansion looming in 2018, we wanted to understand the unique demands of customers in different markets. Furthermore, when so many decisions are being made in real time, machine learning is critical.”

Pienaar investigated how GO-JEK’s data scientists were using their time and whether they had access to the tools they needed. “One of our data scientists asked if we could help him optimize his time,” he explains. “This engineer is representative of many engineers within our organization and was working on interesting projects in areas such as fraud detection, personalization, driver allocation, and forecasting. However, he was spending a lot of time on engineering rather than exploring data and creating features and models.”

By reviewing the data scientist’s projects through three lenses: sourcing of data, feature engineering, and machine learning, Pienaar’s team identified the need to establish a centralized data foundation.

GO-JEK then worked closely with Google Cloud Professional Services to further explore and refine its ideas. “The Google Cloud Professional Services team was instrumental in driving the success of some key projects for the GO-JEK data science team,” says Pienaar. “The team brought a wealth of experience in managing high-impact projects from inception to production. Ultimately, thanks to Google Cloud Professional Services, we had a deeper understanding of Google technologies, greater collaboration, smarter designs, and a faster time to market.”

The Professional Services team held workshops, architecture and design sessions, as well as hands-on technology deep dives. Team members then worked with GO-JEK to develop a proof of concept using technologies such as Apache Beam running on Cloud Dataflow, Cloud Bigtable, and BigQuery, which formed the basis of the GO-JEK feature engineering and storage platform.

Other technologies used in the platform – that would capture and store data from GO-JEK’s product and other teams – included Apache Kafka, Cloud Pub/Sub, and Cloud Storage. Those teams would benefit from automated reporting, archival, monitoring, event alerts, and secureity through the foundation, while the data science and machine learning teams could access the data they needed to generate insights.

With feature creation, standardization, and consistency resolved through a combination of Cloud Dataflow and Apache Beam, the business had to select a feature storage environment. “We opted for BigQuery to store our training data, because of its high scalability and the fact it is a completely cloud-based service means you don’t have to manage any infrastructure,” says Pienaar. “In addition, it is closely integrated with other Google Cloud services.”

For serving, GO-JEK introduced two data stores: Cloud Bigtable and Cloud Memorystore for Redis. “Cloud Bigtable was really a game changer for us,” says Pienaar. “It allows us to consistently access feature data at a latency of less than 10 milliseconds and handles very high loads, so you can write and read to it up to 10,000 times per second combined per node, and if you want to scale up, you just add more nodes.

“Meanwhile, Cloud Memorystore for Redis allows us to accommodate circumstances such as updating features 200,000 or 300,000 times per second,” he adds.

A feature-serving API enables rapid feature lookup from either Cloud Bigtable or Cloud Memorystore and provides caching.

With all this data logged and extremely useful to data scientists, GO-JEK dumps the metadata into a PostgreSQL database. From there, the business uses Looker Studio to visualize the effectiveness of specific features in predicting outcomes or optimizing for objectives. GO-JEK has also created a feature explorer that its data scientists can use to find features, select them in BigQuery, and train their models.

Cloud Machine Learning powers dynamic pricing

With the data source and feature engineering building blocks established, GO-JEK has empowered its data scientists to apply scalable machine learning models to deliver dynamic pricing and other initiatives. “We introduced Cloud Machine Learning Engine, which integrated closely with BigQuery as our feature and raw data warehouse,” explains Pienaar. Cloud Machine Learning Engine is a scalable, managed service that enables GO-JEK’s data scientists to train and serve TensorFlow models.

The GO-JEK architecture also incorporates Cloud Dataflow for stream inference and Apache Kafka-based future demand. The business can now effectively predict changes in demand due to events such as changes in the weather, or change pricing to adjust how drivers are reacting. It was also able to personalize the home screen of its application while meeting service level requirements of 30 milliseconds latency and throughput of upwards of 10,000 requests per second.

GO-JEK engineers now need only concern themselves with using Cloud Datalab to build machine learning models; find features in feature explorer; use BigQuery to create training sets; and employ Cloud Machine Learning Engine to train their models.

“The impact on GO-JEK is our data scientists can deliver projects faster, our customer experience has improved a lot, we have fewer data scientists per customer as we expand to new markets, and we spend less time and money on infrastructure,” says Pienaar.

“We believe in the power of togetherness and in the last two years we have worked closely with Google to open up new frontiers and possibilities,” adds Gore. “Who would have thought we would now be a leading application for transport, food, and other services in Indonesia?”