Skip to content

Rapid7 ICS Bot Deployment

Background

The Rapid7 InsightCloudSec (ICS) platform supports a broad range of Cloud Service Providers. Currently, TruQuery's Public Storage Validation API supports Amazon Web Services (AWS), Microsoft Azure, and the Google Cloud Platform (GCP).

ICS becomes aware when a new cloud storage container is created, or the configuration of an existing storage container is modified. This is obtained either through traditional API driven harvesting or through Event Driven Harvesting (EDH), depending on which method you have configured for each of the cloud provider(s) within your ICS deployment.

Establishing a bot that leverages the signal of creation or change creates the ability to send a payload containing the base cloud provider, and storage container name for public read access validation.

ICS Bot Configuration

We have created a .json bot template that can be imported directly into the ICS Bot Factory using the Import Template functionality of the UI.

You can download the TruQuery Public Storage Bot Configuration at NephoSec's Public Code Repository.

Specific information about each of the bot configuration options are outlined below.

Additional information about creating ICS bots using templates can be found on the ICS Document Site.

About Bot

Default values are provided for the Name, Description, and Category. Modify to meet the naming conventions of your organization.

Resource Type & Groups

Resource Types

The resource type must be Storage Container and additional resource types are not supported for this TruQuery validation.

Groups

The provided .json template lists by default the ICS system badges for AWS, Azure, and GCP which will scope the bot to all Cloud accounts known by ICS for these providers.

Like any other bot, you can remove the provided badges and scope the bot to individual cloud accounts, or organization specific badges.

Note - See Enabling & Running the Bot below about cautions during initial bot deployment.

Query Filters

The provided .json template utilizes the filter "Always Match". We recommend using this filter to ensure all storage containers are validated when they are created or modified.

If desired, other filters such as Resource Name Regular Expression (Regex), or Resource Contains Tag Key can be used instead for initial testing, or if a limited scope of validation is desired across multiple accounts.

Actions

The bot leverages the action Post Request To URL. You can learn more about acceptable payload fields/values on the Public Storage API page for CURL run testing, but by using the ICS platform, you have the advantage of leveraging Jinja2 templating.

Web URL to post a payload to:

https://api.nephosec.com/v1/public-storage

Payload to post to the specified URL:

The provided .json bot template contains the following API payload that will require customization:

{
    "Company": "<Insert Your Company Name>",
    "Name": "{{resource.name}}",
    "Provider": "{{resource.serialize(indent=2, jmespath="common.cloud")}}",
    "Account": "{{resource.serialize(jmespath="common.account_id")}}",
    "Email": ["{{resource.get_tag_value("Owner", case_insensitive=True)}}", "email1@domain.com", "email2@domain.com"]
}
  • Company: The payload value inside the quotes should changed to your company name.
  • Email: The payload value(s) inside the quotes should be changed to email addresses that want the validation results to be sent to. Email addresses can be added or removed as desired.

Note the email entry: "{{resource.get_tag_value("Owner", case_insensitive=True)}}". This entry is using Jinja2 templating to extract the value of the tag key "Owner" if it exists on the cloud storage resource. The tag key "Owner" can be changed to reflect the tagging strategy of your organization such as "TechnicalOwner". This entry format can be repeated for additional tag keys that may be present on the cloud resource. If the tag key and/or value are missing or malformed the API will handle the error and skip the address.

Headers to post to the specified URL:

The provided .json bot template contains the following headers added to the API payload that requires customization:

{
    "accept": "application/json",
    "x-api-key" : "NephoSec Provided API Key"
}
  • x-api-key: The value inside the quotes should changed to the API key provided by NephoSec. Contact support@nephosec.com if your key expires, becomes lost, or to request rotation.

Seconds before timeout:

It is recommended that this value be retained as 60 seconds.

Maximum Number of Attempts:

It is recommended that this value be retained at 3 attempts.

Skip Previously Identified Resources:

It is not recommended that this be selected to ensure that you are notified again if attempts are made to secure the resource and these changes have not remediated the public exposure.

Note - If desired you can add the bot action Mark Resource Non-Compliant for tracking. This will not impact the bots effectivity as long as "Skip Previously Identified Resources" is not selected.

Run Options

The provided .json bot template uses the following reactive hook-points which cause the bot to fire:

  • Resource Created
  • Resource Modified

These two hook-points provide the most coverage without the having to be run on a schedule.

Enabling & Running the Bot

Once you have modified the configuration of the bot to the needs of your organization, save the bot. Upon saving the bot will remain in the Paused state. To place the bot into service promote the bot state to Enabled.

Once enabled the bot will fire against all newly created storage containers and containers where the configuration has changed going forward.

CAUTION - If the bot is run "On-Demand" it will validate all storage containers contained within the cloud accounts scoped to the bot.

This is very helpful to validate all existing storage containers, but for very large cloud deployments running the bot on-demand can place stress on the ICS platform and should be run during off-hours when other activities are at their lowest.

This bot has been tested against over 25,000 storage containers during off-hours in a single run without issue and took ~60 minutes to process all results.

Bot Monitoring:

Each time the bot runs an entry will be placed in the Scheduled Events tab of the bot. Any failed events will show a red exclamation point icon under the Last Run column. Clicking the red icon will reveal the error. The bot configuration should be checked and the individual event rescheduled to be run.

All successful bot runs will be greeted with green check icon under the Last Run column. Clicking on the green icon should show the following result:

{{"content": <YOUR PAYLOAD>, "code": 200, "reason": "OK"}

Trust But Verify... Congratulations! You are now ensuring all storage containers are deployed securely and remain that way.