Our Platform Engineering Project — Collaborative Innovation Accelerator [CIA]

Introduction

This article showcases a project I and my team developed during a company-wide hackathon, detailing the origins of our idea and its implementation. The project later earned recognition by winning an external DevOps award.

Origins

Startups are designed to optimize for fast growth, but with this rapid expansion come inherent challenges. To stay ahead, engineering teams constantly experiment and implement new features. This “move fast and break things” philosophy results in an increasing number of features being shipped, necessitating a growing team to support this rapid development.

As teams expand, so do the complexities of management. Team division and reorganization become frequent, and the number of verticals and products soon begins to multiply. However, not everything can scale at the same rate. A prime example is the environment where new applications are tested and deployed. Most startups operate with a single development, staging, or pre-production environment.

Ideation

We thought, what if we could find a way to increase the number of environments while keeping costs and management complexity under control?

This idea could help engineering teams work faster by allowing parallel deployment and testing in multiple environments. This became the core idea of our project. We aimed to enhance team collaboration, encourage experimentation, and, by extension, foster innovation. Additionally, we wanted to accelerate the entire development process and reduce the time to production.

Building the CIA

The idea was attractive enough for us to proceed. We brainstormed how a platform like this would work, what we should aim for first, and what the developer experience should be. We settled on a few key principles: simplicity, robustness, and feasibility within our 14–15 hour time constraint.

Initial Setup

After the design phase, we had to build a few essential components beforehand:

• Networking and DNS: These were critical because we wanted the CIA to deliver DNS records that allowed developers to make calls to each backend service they requested. We created an ALB Ingress Controller and a small automation to create DNS records in Route 53. We quickly wrote a Helm chart to create the ingresses for us based on the application’s name. This took a lot of time as we had to set up an EKS cluster for our platform too!
• Full-Fledged Vault Deployment: Surprisingly, the beginner’s guide to Vault was enough to get this up and running with a few tweaks. Vault would be used to inject environment variables securely.
• MongoDB Operator Setup: Setting up the MongoDB operator was a significant challenge since none of our team had done it before. It became a valuable learning experience and a tough hurdle to overcome.
• Vault Integration: We faced some struggles integrating Vault with our system, but we eventually figured it out.

The Basic Design

The design was straightforward:

• User Interface: Developers would log in to a basic HTML page with a form. This form asked which microservices the developer wanted to deploy and how long they intended to use the environment.
• Cost Estimator: The HTML page also acted as a cost estimator, giving an estimated cost of running the environment for the specified hours.
• Backend Automation: The backend APIs would read the developer’s input and create several Helm charts: One for a local Redis cache.
• One that created a MongoDB Atlas object using an operator to make calls to MongoDB Atlas on our behalf. We kept customizations minimal for straightforward deployment.
• One for each microservice.

Working

A Vault container would inject the newly created Redis and MongoDB hosts into each microservice according to its environment variables.

To manage costs, we planned to create a cron job to delete the environments after the specified number of hours. This cost-conscious approach was crucial to the project’s viability.

Conclusion

This basic yet powerful design aimed to enable developers to work faster, collaborate more effectively, and innovate continuously without being bogged down by environmental constraints. The Collaborative Innovation Accelerator (CIA) was not just a project; it was a significant leap toward a more efficient and innovative development process for startups.

Achievements

The project won the 2nd runner-up prize in the hackathon and was chosen the overall winner in the “Best Implementation In DevOps” category at the 3rd Annual India DevOps Show 2024, Bangalore.

Credits to all the team members:
Tejas Gupta, Nityesh Maheshwari, Shivam Vishwakarma, Vinay Kumar, Achyut, and Neelesh Bharadwaj.