Revamping Infra Request Experience with Self-Service Git Repo
How we crafted an innovative framework that keeps users happy, engineers smiling, and budget in check.
In my previous post about handling VPN access requests by using Slack Bot, I mentioned a lot about self-service repos. However, it was limited specifically to VPN access requests. Now, I will cover the self-service repo in general.
Usually, most of the infra requests were done manually. By just using the cloud provider console, cloud infrastructure engineers can do many things, from provisioning virtual machines, creating cloud storage, creating firewalls, managing VPC, and so on.
But things are different when involving automation for handling infra requests. As a cloud infrastructure engineer, it greatly reduces my time working on the infra requests. Instead, I let the machine do it asynchronously. In that case, we work a lot on building a self-service Git repo for handling infra requests automatically.
Now, let’s dive deep into the process.
What is a self-service Git repo?
This is the term used when users could submit their infra requests independently. The pipeline and our scripts work on behalf of the infra team, eliminating the need for manual handling of these requests. The infra team has built multiple automations as self-service repo to enable this, allowing everyone to make requests at their convenience. With this approach, the infrastructure team can depend on automation to address user requests.
Hooray for more automation!
Why do we implement a self-service Git repo?
The self-service repo is our attempt to solve these issues:
- Give limited access to our cloud infrastructure console. The interface is only accessible from the self-service repo, and users can’t access the console directly. We use Terraform + Gitlab CI to apply the change on behalf of our users.
- Validate changes before we proceed with the MR review and approval in Gitlab.
- Prevent unwanted changes to our cloud infrastructure resources through review and approval steps via MR in Gitlab.
- Establish auditable and accountable changes by using Gitlab CI.
- Create Service Accounts and IAM exclusively for the self-service, avoiding the assignment of permissions to individual users.
Implementing the self-service Git repo
Declarative
The main heart of the self-service repo is basically a script, which can take various forms such as Python script, Terraform provisioner, Ansible playbook, and many more.
We give our users the YAML file as a meta definition based on the agreed-upon format within the team. This YAML file will then be processed by the pipeline and processed as input by the script.
These are examples of our declarative self-service repo:
- GKE Cluster provisioner
- Kubernetes Namespace Management
- GCP Identity Access Management
- GCP Cloud Storage Buckets Management
- Coralogix Alerts Management
- Grafana Dashboard Management
- Gitlab Projects Audit
- and many more
Technically, we keep the history of changed resources within the self-service repo. As long we don’t store any confidential information, we opt for the declarative approach to construct the self-service repo. This way, we can track what kind of changes have been applied to particular resources, and it’s easily auditable.
Imperative
It’s slightly different from the declarative one. Here, users will get a form from Gitlab and specify all required values to be processed by the self-service pipeline.
We build this approach for managing Vault secrets. In this scenario, the pipeline works as an interface to our self-hosted Vault. YAML is not the best option to store the secret because it’s not a security best practice. Therefore, we use the imperative approach to help users store their secrets securely without having them recorded in the self-service repo.
Imperative + Declarative
Now, you might ask whether we can use both approaches? The answer is yes. We came up with this idea when handling VPN access request by using Slack Bot .
Even though we have provided self-service repo for VPN requests, our non-techies colleagues (business, finance, project managers, etc.) encountered some technical issues. They don’t know how to use Git, and using the Gitlab repo page for submitting changes and MR was hard to follow and took much time. This approach also blew up our budget as non-techies colleagues must be counted in the Gitlab subscription.
Our manager then proposed the idea of using Slackbot as an additional interface for the self-service repo. This way, all users (techies or non-techies) can specify all required parameters to the Slack Bot. Behind the scenes, the pipeline will edit or create YAML file on behalf of the user. Now, users don’t need to use Git or Gitlab Editor for VPN access request submissions. Moreover, we can save a lot of money by reducing the number of Gitlab subscriptions.
So, by building the combination of imperative and declarative through Slack Bot, we tackled two major things. First, accessibility for all users to the self-service VPN access request. Second, cost optimization for Gitlab subscription.
Additionally, we separate the repo self-service repo and the scripts used for the self-service repo. This way, the inner workings remain hidden behind the scenes, like a secret sauce reserved for the chef’s eyes only.
The good parts
Naturally, our team wouldn’t embark on any project without expecting a substantial positive impact. So far, with this self-service implementation, we’ve seen improvements in:
- Reduced Gitlab user subscription cost
- Cloud infrastructure engineers can focus more on their projects than manually handling various infra requests. Now, most of the infra requests are fulfilled automatically.
- Developers and business teams alike can raise infra requests through particular self-service Git repos based on their needs.
- Automatically removing user access in VPN and Gitlab who left the company (compliance to ISO 27001:2013)
- Increase security by adding the least privilege principle layer for developers or business teams who want to provision their infra resources. We avoid giving them direct access to GCP Console (compliance to ISO 27001:2013)
- Strengthening security by adding logging and traceability. We can now audit every change on infra, which is stored on many self-service repos (compliance to ISO 27001:2013)
- More streamlined processes. Take our MongoDB user management self-service repo, for instance. When creating a MongoDB username through the self-service repo, the credentials are seamlessly stored in Vault automatically. This approach helps us combat unsecured practices like sharing passwords via email or Slack DM (compliance to ISO 27001:2013)
- We have a backup of our infra resources just in case of unexpected incidents (compliance to ISO 27001:2013)
- and many more
The bad parts
Once again, there is always room for improvement, and we encountered some challenges during the development of the self-service repo:
- We need to provide simple and better README as instruction for our users — the business team, developers, or even the infra team themselves.
- When the self-service encounters issues, resolving them can take time, potentially causing delays for our users in getting their requests fulfilled.
Aside from the two I mentioned above, we haven’t identified any other disadvantages. We are still working on resolving the identified issues.
Conclusion
Building a self-service Git repo empowers both developers and non-techies colleagues to independently and easily raise infra requests. I believe this approach benefits both the cloud infrastructure team and users.
Special thanks to Platform and Cloud Infrastructure teams for inspiring me to write this article.
Looking for fresh inspiration and practical solutions for your tech projects? Our engineers have you covered. Follow us to stay updated!
