In order to continue to rely on policies we need to ensure that we are able to handle edge cases and make sure any changes do not result in a regression. To do that the most common way is to have a suite of test cases and ensure the output matches the expected output for specific inputs.
High level overview of Policy Testing
- Create specific resources in provider that should lead to a known outcome of a check.
- Store configuration in database (
- Store database configuration and known good output of check in local filesystem.
- Confirm that control is still valid by ensuring that each configuration case continues to generate the same output.
What is a snapshot?
a snapshot is a partial copy of the database as it relates to a specific policy. This means that all relevant tables are exported to a local directory in order to be able to consistently replicate a specific scenario.
Setting up a Snapshot
To ensure that you are grabbing real data you will want to configure your resource to be in a specific configuration (can either be in a passing or failing). Once it is ready run:
This will ensure that the database you are about to snapshot will have valid data.
Storing a Snapshot
cloudquery policy snapshot <selector/to/specific/check> <output/directory>
cloudquery policy snapshot ../aws//foundational_security/apigateway/ApiGateway.2 test/snapshot-data
Prior to committing to a public repository: validate that there is no sensitive information (account IDs, credentials or PII) in the CSV
When working with the cloudquery-policies/aws repository you can use the script found in the tooling directory:
go run tools/find-sensitive-strings.go
The test running utility takes a relative or absolute path to a policy and a path to a directory where test snapshots are stored. The path of the test matches the selector for a corresponding check in the policy. If the results of the check do not match the expected results the test run will end immediately
*Note: At this time it is not possible to run only a single test, you have to run all tests in the entire directory
Do not run tests against a production database as the testing processes is destructive in terms of truncating tables and dropping foreign keys.
We recommend spinning up a local postgres instance on docker and then deleting the instance after the test run
cloudquery policy test <selector/to/root/policy> <output/directory>
cloudquery policy test ../aws test/snapshot-data