We all know how vital shifting security left is and how engineering needs to work more closely with security. According to the Cloud Security Alliance (CSA), 70% of security professionals and engineering teams struggle to “shift left,” with many unable to recognize the formation of anti-patterns and understand/appreciate the wider impact to development, cost, governance, culture, etc.
Recognizing the Paradigm Shift
Engineering teams that implement DevSecOps practices and automate security tooling will discover security risks earlier, saving developers time, accelerating release cycles, and shipping more secure and compliant code.
As I have previously discussed, dependency management is an important part of securing your software supply chain. This post covers how to roll out Dependabot internally within your company to keep your dependencies up to date.
NB: There are other tools such as Renovate and Snyk, but as Dependabot is native to the GitHub platform I am focusing the talk on this tool.
Keeping your dependencies up to date is one of the easiest ways to keep your systems secure. The issue of supply chain security has become increasingly obvious in the past number of years, from the malicious flatmap-stream package to the most recent log4shell vulnerabilities. Dependabot will alert developers when a repository is using a software dependency with a known vulnerability. By rolling out Dependabot internally to all of your repositories, you can measure, and significantly reduce, your usage of software dependencies with known vulnerabilities.
Everyone will remember the security breach at Experian in 2017. Hackers stole the personal details of 143 million Americans from the Equifax credit report company after exploiting a security flaw in the Apache Struts framework. That is why it is so important to keep track of your dependencies and any known vulnerabilities.
OWASP published last year their top ten, and vulnerable and outdated components moved up to sixth position. Every organization must ensure an ongoing plan for monitoring, triaging, and applying updates or configuration changes for the lifetime of the application or portfolio. There are over 20,000 common vulnerabilities and exposures (CVEs) discovered per year in open-source and third-party code.
Empowering developers to be security minded
According to the SANS 2022 DevSecOps Survey: Creating a Culture to Significantly Improve Your Organization’s Security Posture, “management buy-in” was the number one factor contributing to DevSecOps security programs’ success.
What is Dependabot?
There are a number of guiding principles when evaluating tools and designing a rollout plan. For example, Does the security benefit of this new process outweigh the impact on engineering teams? How do we roll this out incrementally and gather feedback? What are our expectations for engineers, and how do we clearly communicate these expectations?
For Dependabot in particular, some of these questions are easy to answer. Dependabot is a native feature of GitHub, meaning, that it integrates with your engineers’ current workflows on GitHub.com. By better tracking the security of your software supply chain, you will keep your software secure, which outweighs any potential impact on engineering teams.
When Dependabot raises pull requests, these pull requests could be for security or version updates:
- Dependabot security updates are automated pull requests that help you update dependencies with known vulnerabilities.
- Dependabot version updates are automated pull requests that keep your dependencies updated, even when they don’t have any vulnerabilities. To check the status of version updates, navigate to the Insights tab of your repository, then Dependency Graph, and Dependabot.
- Dependabot security updates may include compatibility scores to let you know whether updating a dependency could cause breaking changes to your project.
Caveats
- Dependabot alerts - Owners of private repositories, or people with admin access, can enable Dependabot alerts by enabling the dependency graph and Dependabot alerts for your repositories.
- Secrets - When a Dependabot event triggers a workflow, the only secrets available to the workflow are Dependabot secrets. The simplest solution is to store the token with the permissions required in an action and in a Dependabot secret with identical names.
- Semantic versioning – don’t trust that every third party dependency aligns with this!
- Vulnerability database - GitHub's security features do not claim to catch all vulnerabilities and malware. They actively maintain GitHub Advisory Database and generate alerts with the most up-to-date information. However, they cannot catch everything or tell you about known vulnerabilities within a guaranteed time frame. These features are not substitutes for a human review of each dependency for potential vulnerabilities or any other issues, and they recommend consulting with a security service or conducting a thorough dependency review when necessary.
Beware of dependency fatigue
You may have heard of alert fatigue before when you are on a support roster. Well, the same can be said for dependency management: I call it dependency fatigue! The last thing you want is for teams to not configure dependabot properly and start to ignore the PRs and alerts created or turn them off completely. So what can we do to help alleviate this? Below I provide some tips and tricks I have used in the past and some sample configs.
Tips
- Design config with your team - Get the team to review the configuration and agree on the setup.
- Understand what
package-ecosystem(s)to enable (docker, maven, actions, terraform etc) - Add
labelsto your PRs to allow easier filtering - Prioritise the order and schedule
- Assign ownership to who is on call
- If your dependencies are very outdated, you might want to start with a daily schedule until the dependencies are up-to-date, and then drop back to a weekly schedule.
- Review the changelog (look for any breaking changes announced) and make sure checks pass on your branch
- It's good practice to have automated tests and acceptance processes in place so that checks are carried out before the pull request is merged.
- Test and learn! 😅
Sample config
You can place this file dependabot.yml under your .github directory in your repo.
This example below showcases a repo that is interested in two eco-systems: docker and actions. I have added in a schedule for docker updates in my local timezone of Melbourne. And I have added some labels. This helps to filter your PRs if you want to focus on specific updates. The open-pull-requests-limit default is 5, so in this instance, I have reduced it to 2.
The next example showcases a sample NodeJS project. Again you are interested in updates from two ecosystems: npm and actions. What this config is also showcasing is how you can ignore certain dependencies. You can see that it ignores packages that start with aws, ignores updates to express and patch updates for all dependencies.
Lastly, is a sample config for a JVM-based project. Here we are covering three package ecosystems as it's a cloud-native microservice. It covers updates to maven, docker and actions.
Below is a sample PR that was created by dependabot based on the config above. You can see it has added the labels of dependencies and maven. It provides the release notes and changelog for you to review.
If you want to know more about the configuration options for dependabot, check out the doc here
Rollout
Dependabot, like other GitHub Advanced Security features, can be enabled for all repositories within an organization from the organization’s administration page. Depending on how you create new GitHub repositories (infra as code anyone??), you can also enable upon creation by default. That way there is no major impact on teams and a staged rollout is not needed. Frequent company-wide comms are recommended.
The aim is to answer, clearly and succinctly, the most important questions in your communications: What are we doing? Why are we doing this? When are we doing this? Lastly, what do I need to do? The last question was key. Make it clear that you are rolling out Dependabot organization-wide to understand your current risk and that, while we encourage service owners to upgrade dependencies, you are not expecting every Dependabot alert to be fixed right away.
Running an engineering workshop on dependency management and dependabot can help educate teams on the importance of this initiative and shifting security left. Showcasing how to manage security and version updates for your services via dependabot will set the foundations and kick-start the conversation. Like any change management framework, it is important to communicate/educate, build the capability and change the mindsets.
Remediation
Once you had Dependabot enabled for all your GitHub repositories, you could measure the general trend of Dependabot alerts across the company. At an org level under the insights tab, you have a bird's eye view of dependencies and open security advisories. You can filter by most critical and then switch your focus from measuring the current state to working with repository owners to upgrade their dependencies. By leveraging insights, you can ensure that you focus your remediation efforts on repositories that are running in production, where a vulnerable dependency could present a risk to your company and your users. As I mentioned earlier, "management buy-in" is extremely important in incorporating dependency management into your security program. Otherwise, you will end up competing with feature delivery and it will fall further down the pecking order. Security trumps customer centricity.
If you have a platform engineering team you could go even further and build or use a nice internal service catalog tool like backstage and hook into GitHub graphQL API to track this. As part of measuring system health and running an internal security program, you can work with your stream-aligned teams to define SLOs and SLIs regarding timeframes to resolve. An example could be:
"Critical and high severity vulnerable dependencies are fixed within 30 days".
It is important to acknowledge that not all Dependabot alerts can be actioned immediately. Instead, assign a realistic grace period for service owners to remediate Dependabot alerts before marking a metric as failing.
Outcomes
The goal here is to reduce the no. of Dependabot alerts over time for your services. But do not regard this as a once-off task. Rather, you need to follow the dependabot alert metrics continuously over time and intervene if you see them trending in the wrong direction. This will help you have a better picture of your system health across the company and prioritise work.
Happy updating! 😀