How to Manage Multiple AWS Profiles For Boto3?

AWS profiles are just named configurations that you can use to organize your credentials across multiple AWS accounts, accounts can be from the same account or multiple accounts, and each profile has its own Access Key, Secret key and other basic user preferences.

Need of Multiple AWS Profiles

When managing multiple projects, each with distinct AWS configurations or handling personal and work-related accounts, organizing and dynamically utilizing credentials becomes crucial. AWS offers a solution by allowing you to configure profiles for each user, ensuring seamless and organized access to different accounts based on specific project requirements. This approach streamlines the process of switching between user accounts, facilitating efficient and secure development across various scenarios.

Simply Multiple profiles allow developers to work on different projects, each with its unique AWS configuration, ensuring separation and security.

Prerequisites

To try the stuff mentioned in this article, you should be ready with the following:

• An AWS account
• Python is installed in your system
• AWS CLI Installed
• boto3 installed

If you haven't set up your system for Boto3 then use the command to install AWS CLI in your Linux system.

sudo apt-get install awscli -y

pip install boto3

AWS Profile Terms

• Access_Key: The AWS Access Key is a unique identifier associated with an AWS account or IAM user. It serves as a credential to access AWS services programmatically through APIs or command-line tools. Access keys are used in conjunction with the AWS Access Key ID for authentication. You can think of it as an username for AWS cli.
• Secret_Access_Key: The Secret Access Key is the corresponding secret key paired with the AWS Access Key ID. It is a sensitive piece of information that must be kept confidential. The combination of Access Key ID and Secret Access Key is used to authenticate and authorize requests to AWS services, ensuring secure access to resources.
• Region: Basically AWS deployed it's data centers across the world, you can choose geographical area i.e., region near to you or your client based on the application requirement, to achieve the low latency, high throughput applications. While configuring AWS profile you can set the default region (this is optional and you can override it anytime)
• Output Format: This is the default output format for the cli, this is an optional config, default is json. Available options are json, yaml, text, table.

Using AWS CLI Profiles with Amazon Braket SDK

When using the Amazon Braket SDK, it typically relies on the default AWS CLI credentials. However, if you want to run your code locally (like on an Amazon EC2 instance), you can set up named AWS CLI profiles. This allows you to assign different permissions to each profile without overwriting your default settings.

1. Set Up a Local AWS CLI Profile

To start, you'll need to create a user and configure your AWS CLI with a non-default profile. For detailed instructions, refer to the official documentation on setting up AWS CLI and using AWS IAM Identity Center. Make sure the user you create has the necessary permissions for Amazon Braket, such as attaching the AmazonBraketFullAccess policy.

2. Create a Boto3 Session Object

Once your profile is set up, you can establish a Boto3 session. Here’s a simple code snippet to do that:

from boto3 import Session 

# Replace `profile` with your CLI profile name
boto_sess = Session(profile_name='profile')

If your API calls require a specific AWS region, you can include that in your session setup:

# Replace `profile` with your CLI profile name and `region` with the desired AWS region
boto_sess = Session(profile_name='profile', region_name='region')

Be sure to choose a region where Amazon Braket is available, such as us-east-1 or us-west-1.

3. Use Boto3 Session with Amazon Braket

Next, you can integrate your Boto3 session with Amazon Braket. Here’s how to initialize a Braket session and create a device:

from braket.aws import AwsSession, AwsDevice

# Create a Braket session using the Boto3 session
aws_session = AwsSession(boto_session=boto_sess)

# Specify the ARN for the desired quantum device
sim_arn = 'arn:aws:braket:::device/quantum-simulator/amazon/sv1'
device = AwsDevice(sim_arn, aws_session=aws_session)

With this setup, you can now submit quantum tasks using the device.run(...) command. All API calls made will use the IAM credentials associated with the CLI profile you specified.

Configure AWS Profile

To configure AWS profile, run the below command

aws configure

This command has an optional argument profile, if you don't pass this argument it will configure for profile name 'default', and this will be your default aws profile.

1. Configure AWS Profile with Name

aws configure --profile <profile-name>

It prompts for Access key, secret key, default region, and output format.

2. Configure Additional Profiles

In the same way as above we can configure multiple profiles by providing the appropriate profile names.

aws configure --profile myprofile2

aws configure --profile myprofile3

3. Accessing Resources with Specific Profile

Note: We're just using an sample IAM boto3 api call for demonstrating the scenario:

We can access the aws resources without specifying the profile like below:

Accessing without profile (here aws sdk will look for default profile, if default profile doesn't exist it will raise an error):

import boto3
client = boto3.client("iam")
response = client.get_user(
    UserName='myname'
)

We can access aws resources by using specific profile like below:

import boto3
session = boto3.Session(profile_name='myprofile2')
client = session.client("iam")
response = client.get_user(
    UserName='myname'
)

Here in the above case, we're configuring the aws profile before initializing the client, so that every client initialized using the configured session will use your profile credentials ('myprofile2' in this case).

Setting a Default Profile in Boto3

Unlike specifying the profile in the session, we can set default profile for the boto3 like below:

import boto3
boto3.setup_default_session(profile_name='myprofile')
client = boto3.client("iam")
response = client.get_user(
    UserName='myname'
)

Here in this case, once you've configured default profile for boto3, so until the end of the code, you need not mention profile again specifically, it will pick the configured profile by default.

Configuring Credentials in Boto3

In Boto3, configuration data is categorized into two main types: credentials and non-credentials. Credentials typically include the following:

• aws_access_key_id
• aws_secret_access_key
• aws_session_token

On the other hand, non-credential configurations involve settings like the AWS region or the addressing style for Amazon S3. For further details on configuring non-credential settings, refer to the Configuration guide.

Boto3 Credential Search Process

Boto3 follows a specific order to locate credentials, stopping once it finds them. The sequence of locations it checks is as follows:

• Parameters passed directly in the boto3.client() method
• Parameters provided when creating a Session object
• Environment variables
• The shared credentials file located at ~/.aws/credentials
• The AWS config file located at ~/.aws/config
• Assume Role provider
• Boto2 configuration file found at /etc/boto.cfg and ~/.boto
• Instance metadata service for EC2 instances with an IAM role configured

1. Passing Credentials as Parameters in Boto3

You can pass credentials directly when creating a client or a Session object. This approach is useful for scenarios such as retrieving temporary credentials with AWS Security Token Service (STS) or loading credentials from an external source, like an OS keychain.

Example of Creating a Client:

import boto3 
 client = boto3.client       
     (     's3',                
            aws_access_key_id=ACCESS_KEY,                   
            aws_secret_access_key=SECRET_KEY,                   
            aws_session_token=SESSION_TOKEN
 )

Example of Creating a Session Object:

import boto3  

session = boto3.Session(    
        aws_access_key_id=ACCESS_KEY,     
        aws_secret_access_key=SECRET_KEY,
        aws_session_token=SESSION_TOKEN 
)

2. Using Environment Variables for Credentials

Boto3 can also retrieve credentials from the following environment variables:

• AWS_ACCESS_KEY_ID: Your AWS account access key
• AWS_SECRET_ACCESS_KEY: Your AWS account secret key
• AWS_SESSION_TOKEN: Your temporary session token (needed for temporary credentials)

You can also use AWS_SECURITY_TOKEN for backward compatibility, but it’s recommended to use AWS_SESSION_TOKEN.

3. Shared Credentials File for Boto3

By default, Boto3 checks the shared credentials file located at ~/.aws/credentials, but this can be changed by setting the AWS_SHARED_CREDENTIALS_FILE environment variable. The file uses an INI format where each section corresponds to a profile. Here’s a simple example

[default]
aws_access_key_id=foo
aws_secret_access_key=bar
aws_session_token=baz

Multiple profiles can be defined in this file:

[default]
aws_access_key_id=foo
aws_secret_access_key=bar

[dev]
aws_access_key_id=foo2
aws_secret_access_key=bar2

[prod]
aws_access_key_id=foo3
aws_secret_access_key=bar3

To use a specific profile, set the AWS_PROFILE environment variable or specify the profile_name argument when creating a Session:

import boto3 
session = boto3.Session(profile_name='dev')
dev_s3_client = session.client('s3')

4. AWS Config File for Boto3

Boto3 can also load credentials from the AWS config file at ~/.aws/config. You can change the default location by setting the AWS_CONFIG_FILE environment variable. The config file has a similar INI format, but profile sections must start with profile:

[default]
aws_access_key_id=foo
aws_secret_access_key=bar

[profile dev]
aws_access_key_id=foo2
aws_secret_access_key=bar2

[profile prod]
aws_access_key_id=foo3
aws_secret_access_key=bar3

5. Assume Role Configuration in Boto3

To use IAM roles, you can configure a profile in the ~/.aws/config file that specifies a role to assume. Boto3 will handle the Assume Role calls to AWS STS automatically. Important settings include:

• role_arn: The ARN of the role to assume
• source_profile: The Boto3 profile with credentials for the AssumeRole call
• external_id: An optional unique identifier for third-party access
• mfa_serial: Optional, for MFA authentication when assuming a role
• role_session_name: Optional, defines the session name
• duration_seconds: The session duration in seconds

Example Configuration:

# In ~/.aws/credentials:

[development]
aws_access_key_id=foo
aws_secret_access_key=bar

# In ~/.aws/config
[profile crossaccount]
role_arn=arn:aws:iam:...
source_profile=development

6. Assume Role with Web Identity

You can also configure Boto3 to assume a role using a web identity provider, such as OAuth 2.0. This is done by specifying the role and the path to the token file in the ~/.aws/config file:

[profile web-identity]
role_arn=arn:aws:iam:...
web_identity_token_file=/path/to/token

This configuration can also be set via environment variables:

• AWS_ROLE_ARN: The ARN of the role to assume
• AWS_WEB_IDENTITY_TOKEN_FILE: The path to the token file
• AWS_ROLE_SESSION_NAME: The name for the session

7. AWS Single Sign-On (SSO)

Boto3 supports AWS SSO, which was introduced in version 1.14.0. You can set up SSO profiles using the AWS CLI V2. After configuration, your SSO profiles will look like this:

[profile my-sso-profile]
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789011
sso_role_name = readOnly

To create a session using an SSO profile:

import boto3
session = boto3.Session(profile_name='my-sso-profile')
s3_client = session.client('s3')

8. Boto2 Configuration

For backward compatibility, Boto3 can also read credentials from the Boto2 configuration file, which can be specified by the BOTO_CONFIG environment variable or found at /etc/boto.cfg and ~/.boto. Only the [Credentials] section is used.

Example

[Credentials]
aws_access_key_id = foo
aws_secret_access_key = bar

9. IAM Roles for EC2 Instances in Boto3

If you run Boto3 on an EC2 instance configured with an IAM role, Boto3 automatically retrieves credentials from the instance metadata service. No additional configuration is required in this case.

Best Practices for Credential Configuration

• Use IAM roles: If you are running on an EC2 instance, always utilize AWS IAM roles.
• Shared credentials file: For interoperability with multiple AWS SDKs (Java, JavaScript, Ruby, PHP, .NET, AWS CLI, Go, C++), use the shared credentials file located at ~/.aws/credentials.

Connecting to Cloudfront Using a Specific AWS Profile

In environments where multiple AWS accounts or roles are in use, it's important to connect to AWS CloudFront with the correct credentials. AWS profiles allow you to manage different sets of credentials easily and securely. By specifying a profile when connecting to CloudFront, you ensure that you're using the right permissions for the job, avoiding potential access errors or unintentional use of another account. This method is particularly useful when working across various environments, such as development, staging, or production, ensuring both security and operational efficiency.

To know more about it you can refer to this article How to Choose an AWS Profile When using Boto3 to Connect to CloudFront

Conclusion

In this article we learnt how to create multiple profiles in aws cli, and access resources using specific profile with different methods. Multiple profiles allow us to work on different projects, each with its unique AWS configuration, ensuring separation and security.