Introduction
During 2024 Profero’s research and incident response teams tracked a trend of cyber-attacks that are based on security misconfigurations and leaking of secrets into the production environment.
In today’s world, developers need to manage a large number of secrets; credentials, API keys, tokens, and passwords, and those secrets are essential for the operation of modern applications. And as the number of these secrets increases, the challenges associated with their secure storage increase as well.
Follow along this short readiness booster blog that will guide you through best practices for secrets management general guidelines.
The Challenge of Managing Secrets
Developers used to store secrets directly in their source-code or within configuration files, this approach would not fit in today’s context of development best practices. In the past, codebases were often kept within isolated environments and had limited access. But now, source-code is shared, reviewed, and deployed across various environments, increasing the exposure of hard-coded secrets.
Security Risks of Hard-Coded Secrets
If a developer commits a file containing secrets to a public repository, those secrets can be easily
- discovered and exploited by malicious actors.
Even within private repositories managed by the company, the rise of stolen
- credentials-based attack increases the risk surface.
Rise of Stolen Credentials-Based Attacks
According to Verizon’s Data Breach Investigations Report (DBIR), above 40% of the attacks today are stolen credentials based. When a developer’s personal computer is compromised, an attacker can gain access to their Git accounts. Then, they can exfiltrate sensitive information, including improperly stored secrets. This can lead to harsh consequences, including unauthorized access to critical company resources, data breaches, and much more.
Best Practices for Secrets Management General Guidelines
Secrets Management Tools
Choosing a secrets management tool such as HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault, will provide a secure storage and access controls for sensitive information.
Access Controls and Auditing
By implementing strict access controls and audits of secrets, the security team can ensure that only authorized employees and systems have access, and that everything is logged.
Automated Secret Rotation
Rotate secrets and credentials regularly to minimize the impact of any potential exposure. Automated rotation mechanisms can help ensure that this process is consistent and reliable.
Secret Scanning
Secret Scanning is a useful tool for security engineers as it gives them the power to detect secrets without interfering or changing the R&D workflow in a drastic way. Integrating tools to scan secrets can be implemented inside the CI/CD pipeline and even without
Complete Removal of Introduced Secrets
When a secret is introduced to the codebase, a complete removal of it is necessary.
It is recommended to train the security team on how to do a surgical removal of the secret in advance as it might be a bit complex. If not done properly, the secret might still exist in the git history and attackers could find it using scanning tools.
Integrating Secret Scanning in CI/CD Pipelines
Secret scanning is the automated detection of secrets leaking in to the codebases and configuration files. The integration of secret scanning tools within CI/CD pipelines provides a proactive approach to detect and address these exposures early in the development cycle.
Choose a Secret Scanning Tool
There are several tools available for secret scanning, including open-source solutions like TruffleHog, GitLeaks, and commercial offerings such as GitGuardian and Aqua Security. Select a tool that fits the needs of your organization in terms of detection capabilities, integration ease, and support.
Integrate with Version Control Systems
Secret scanning tools should be integrated with your version control system (VCS) to scan repositories for exposed secrets. For instance, tools like GitGuardian can be configured to monitor GitHub, GitLab, or Bitbucket repositories continuously.
Set Up Scanning in CI/CD Pipelines
Modify your CI/CD pipeline configuration to include secret scanning as part of the build process. This can be done in several different approaches:
Adding a secret scanning step in your pipeline definition files (e.g., Jenkinsfile,
- .gitlab-ci.yml, or GitHub Actions workflow files).
In a case where adding steps to the pipeline is out of scope, security engineers can approach the issue with a periodical secret scanning solution. This can be done by either implementing an existing solution such as GitGuardian and Snyk, or by writing a script and running as a cronjob to scan across the entire org and
- send alerts.
Handling Exposed Secrets in Source-code
When a developer pushes a commit and introduces a secret to the repository, immediate action is required to mitigate the risk. If the branch has been merged into the main/master branch a surgical removal of the commit is necessary from the git history, If it’s an unmerged branch you can just delete the branch.
If the Secret is in the Main/Master Branch
Removing the Commit Surgicaly: Use tools like BFG Repo-Cleaner and git-filter-repo to remove the specific commit containing the secret from the repository’s history.
That way you can ensure that the secret is not only removed from the current code but
- also from the history where it could be accessible.
Rotate the Secret: Revoke and regenerate the compromised secret.
- Update the application and services to use the new secret.
Example using BFG Repo-Cleaner
Removing specific files:
bfg --delete-files '<filename-or-pattern>'
Replacing specific texts:
bfg --replace-text '<password-or-key>'
Cleanup:
git reflog expire --expire=now --all && git gc --prune=now –aggressive
If the Secret is in an Unmerged Branch
Delete the Branch: Remove the unmerged branch to prevent the secret from being merged into the main codebase. Ensure that any references to
- the secret are removed before any further actions are taken on the branch.
Example
To delete the local branch:
git branch -D <branch-name>
To delete the remote branch:
git push origin --delete <branch-name>