Secrets leakage – A rising threat. Development Practices to Safeguard Your Secrets

Profero IRT

July 24, 2024

Share this post

## Introduction During 2024 Profero’s research and incident response teams tracked a trend of cyber-attacks that are based on security misconfigurations and leaking of secrets into the production environment. In today’s world, developers need to manage a large number of secrets; credentials, API keys, tokens, and passwords, and those secrets are essential for the operation of modern applications. And as the number of these secrets increases, the challenges associated with their secure storage increase as well. Follow along this short readiness booster blog that will guide you through best practices for secrets management general guidelines. ## The Challenge of Managing Secrets Developers used to store secrets directly in their source-code or within configuration files, this approach would not fit in today’s context of development best practices. In the past, codebases were often kept within isolated environments and had limited access. But now, source-code is shared, reviewed, and deployed across various environments, increasing the exposure of hard-coded secrets. ## Security Risks of Hard-Coded Secrets If a developer commits a file containing secrets to a public repository, those secrets can be easily * discovered and exploited by malicious actors. Even within private repositories managed by the company, the rise of stolen * credentials-based attack increases the risk surface. ## Rise of Stolen Credentials-Based Attacks According to Verizon’s Data Breach Investigations Report (DBIR), above 40% of the attacks today are stolen credentials based. When a developer’s personal computer is compromised, an attacker can gain access to their Git accounts. Then, they can exfiltrate sensitive information, including improperly stored secrets. This can lead to harsh consequences, including unauthorized access to critical company resources, data breaches, and much more. # Best Practices for Secrets Management General Guidelines ## **Secrets Management Tools** Choosing a secrets management tool such as HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault, will provide a secure storage and access controls for sensitive information. ## Access Controls and Auditing By implementing strict access controls and audits of secrets, the security team can ensure that only authorized employees and systems have access, and that everything is logged. ## Automated Secret Rotation Rotate secrets and credentials regularly to minimize the impact of any potential exposure. Automated rotation mechanisms can help ensure that this process is consistent and reliable. ## Secret Scanning Secret Scanning is a useful tool for security engineers as it gives them the power to detect secrets without interfering or changing the R&D workflow in a drastic way. Integrating tools to scan secrets can be implemented inside the CI/CD pipeline and even without ## Complete Removal of Introduced Secrets When a secret is introduced to the codebase, a complete removal of it is necessary. It is recommended to train the security team on how to do a surgical removal of the secret in advance as it might be a bit complex. If not done properly, the secret might still exist in the git history and attackers could find it using scanning tools. # Integrating Secret Scanning in CI/CD Pipelines Secret scanning is the automated detection of secrets leaking in to the codebases and configuration files. The integration of secret scanning tools within CI/CD pipelines provides a proactive approach to detect and address these exposures early in the development cycle. ## Choose a Secret Scanning Tool There are several tools available for secret scanning, including open-source solutions like TruffleHog, GitLeaks, and commercial offerings such as GitGuardian and Aqua Security. Select a tool that fits the needs of your organization in terms of detection capabilities, integration ease, and support. ## Integrate with Version Control Systems Secret scanning tools should be integrated with your version control system (VCS) to scan repositories for exposed secrets. For instance, tools like GitGuardian can be configured to monitor GitHub, GitLab, or Bitbucket repositories continuously. ## Set Up Scanning in CI/CD Pipelines Modify your CI/CD pipeline configuration to include secret scanning as part of the build process. This can be done in several different approaches: Adding a secret scanning step in your pipeline definition files (e.g., Jenkinsfile, * .gitlab-ci.yml, or GitHub Actions workflow files). In a case where adding steps to the pipeline is out of scope, security engineers can approach the issue with a periodical secret scanning solution. This can be done by either implementing an existing solution such as GitGuardian and Snyk, or by writing a script and running as a cronjob to scan across the entire org and * send alerts. # Handling Exposed Secrets in Source-code When a developer pushes a commit and introduces a secret to the repository, immediate action is required to mitigate the risk. If the branch has been merged into the main/master branch a surgical removal of the commit is necessary from the git history, If it’s an unmerged branch you can just delete the branch. ## If the Secret is in the Main/Master Branch Removing the Commit Surgicaly: Use tools like BFG Repo-Cleaner and git-filter-repo to remove the specific commit containing the secret from the repository’s history. That way you can ensure that the secret is not only removed from the current code but * also from the history where it could be accessible. Rotate the Secret: Revoke and regenerate the compromised secret. * Update the application and services to use the new secret. ### ### Example using BFG Repo-Cleaner Removing specific files: ```vim bfg --delete-files '' ``` Replacing specific texts: ```powershell bfg --replace-text '' ``` Cleanup: ```plain git reflog expire --expire=now --all && git gc --prune=now –aggressive ``` ## If the Secret is in an Unmerged Branch Delete the Branch: Remove the unmerged branch to prevent the secret from being merged into the main codebase. Ensure that any references to * the secret are removed before any further actions are taken on the branch. ### Example To delete the local branch: ```bash git branch -D ``` To delete the remote branch: ```maxima git push origin --delete ```

Share this post

Profero IRT

Profero Incident Response Team - From the trenches insights from IRT

Solution

Get Started Under Attack?

About

Company Careers

Resources

Blog Github Trust Portal

Contact

X/Twitter

Related posts