
Today, I had the opportunity to learn about scaling secrets management using Vault. While I won't go into detail about our previous approach to save some embarrassment, I will say that it didn't align with the core principles of Vault. However, one of my colleagues addressed security risks by implementing Vault's dynamic secret features in our cicd pipelines. This involved adding various authentication methods and configuration bits, which gave me a better understanding of how Vault was intended to be used. As I continue to learn more about it, I am continually impressed by the thoughtful design behind it. Kudos to the HashiCorp team, as well as my team for digging deeper into Vault!

#### Jenkins

We created a test CICD pipeline consisting of the following steps:

- login to Vault with github method (github token)
- sleep for 10 seconds (to wait until queued api calls are processed)
- make a sample api call to AWS (e.g. aws ecr describe-repositories)
- print out the result for debugging purposes
- revoke a lease_id (part of the output from the previous step) associated with temporary aws credentials
- make another api call to verify that it errors out (403)

#### AWS

- create an IAM policy allowing iam-related actions
- create a dedicated user that can create temporary users (let's call it `vault-root-user`) and attach the policy created above
- create another IAM policy allowing certain actions on certain resources (e.g. read/write access to ECR and S3 for a jenkins agent)
- create a group and attach the policy above to group temporary users that will be created/deleted by jenkins agent

#### Vault

- enable the AWS secrets engine and configure the credentials of the vault-root user to connect to AWS
- configure a vault role that maps to a set of permissions in AWS as well as an AWS credential type. Alternatively, an iam group (e.g. `iam_groups=cicd`) can be supplied instead of a set of permissions
- create a policy dedicated to github auth method, allowing limited access (e.g. read/list) to the `/aws/creds/*` path
- enable the GitHub auth method and attach a token policy
- update the existing default policy by adding another rule that allows revoking a lease_id

```
path "sys/leases/revoke/*" {
    capabilities = ["update"]
}
```

Key outcomes from the above include that AWS credentials have temporary, limited access, so there's no need to worry about any potential secrets being leaked once the pipeline is finished.

Additionally, we can explore the integration of Vault and Terraform, both of which are HashiCorp products, by codifying the Vault configurations. This powerful combination would amplify the benefits of Infrastructure as Code (IaC). I'm excited to give it a try in the next round!


Vault in CICD


One of my teammates took an initiative to improve the way secrets were managed in the source code. He brought up HashiCorp Vault, which is a secret management solution that enables developers to handle the storage and retrieval of sensitive information with ease.

My responsibility was to set up Vault in one of the existing Amazon EKS clusters. While integrating Vault into the system, I encountered some configuration issues, which forced me to learn deeper about many different components of Vault. The more I dived into it, the more I realized how useful and powerful it was.

So, let's start with a high-level overview of vault architecture. It is roughly composed of four components:

- **Vault server**: a vault server can be deployed with a helm chart, which would generate a set of required resources easily and quickly.
- **Storage backend** (e.g. Consul, MySQL, etc): MySQL (v8.0.30) docker image is used. A script containing sql statements in the docker-entrypoint-initdb.d file is mounted, such that initial set up is executed when a MySQL container starts up.
- **Authentication backend** (third-party services such as AWS, GitHub, LDAP, Okta or built-in AppRole): AppRole auth method is used.
- **Client**: users, machines, applications that consume secrets

In order to retrieve a secret from vault, a token is required. This token has a policy attached to it, and the policy specifies granular access (which action on which path is allowed for this particular client). For instance, one policy would state that only "read" action is allowed at /secret/data/webapp path.

To acquire a token, a machine or an application needs to authenticate with vault against a given auth method. One example of an auth method is AppRole. To authenticate with AppRole, the following credentials should be provided:

- `ROLE_ID`: the role id is like a static username that identifies an application. it can be shared across multiple instances of the same app, and can be embedded into a machine image.

- `SECRET_ID`: the secret id is like a dynamically generated password that is associated with a particular role id.

The acquired token can then be provided to make API requests to vault server. Best practice regarding the AppRole authentication is to use two different channels to provide role id and secret id, but never both. For instance, we can obscure the role id from jenkins (by embedding into a machine image), but allow jenkins to deliver a wrapped secret id to the application.

**References**

- https://www.vaultproject.io/docs/what-is-vault
