diff --git a/s3-bedrock-data-automation-sam/README.md b/s3-bedrock-data-automation-sam/README.md new file mode 100644 index 000000000..dba03e6fa --- /dev/null +++ b/s3-bedrock-data-automation-sam/README.md @@ -0,0 +1,82 @@ +# Extract structured data from documents with Amazon Bedrock Data Automation + +This pattern turns files uploaded to Amazon S3 into structured JSON using Amazon Bedrock Data Automation (BDA) and AWS Lambda, with no machine-learning code. Drop a document (or image, video, or audio file) into the `input/` prefix and BDA writes the extracted result to the `output/` prefix. + +Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/s3-bedrock-data-automation-sam + +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## How it works + +``` + upload file ObjectCreated invoke async + to input/ --> Amazon S3 -------------> AWS Lambda ------------> Bedrock Data Automation + ^ (start job) (managed extraction) + | | + +---------------- structured JSON to output/ -------------+ +``` + +- A file uploaded to the `input/` prefix raises an S3 `ObjectCreated` event that triggers the Lambda function. +- The function calls the BDA runtime `InvokeDataAutomationAsync` API with the input file, an output location, and the Bedrock Data Automation project. It is fire-and-forget - it only starts the job. +- Bedrock Data Automation reads the file, runs the managed extraction defined by the project, and writes structured JSON to the `output/` prefix. +- The project (`AWS::Bedrock::DataAutomationProject`) is a native CloudFormation resource, so the whole pipeline is infrastructure as code. The included standard configuration returns each document as Markdown text (including tables) plus a generative summary. + +## Requirements + +- An AWS account with permissions for Amazon S3, AWS Lambda, and Amazon Bedrock Data Automation. +- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) v2, recent enough to include the `bedrock-data-automation` and `bedrock-data-automation-runtime` commands. +- [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html). +- A Region where Amazon Bedrock Data Automation is available (for example, `us-east-1`). + +## Deployment + +```bash +sam build +sam deploy --guided +# - Stack Name : s3-bedrock-data-automation +# - AWS Region : us-east-1 (or a Region where BDA is available) +# - DataBucketName : a globally-unique bucket name, for example my-bda-demo- +# - Allow SAM CLI IAM role creation : Y +``` + +Note the `DataBucketName` output after deployment. + +## Testing + +Upload any supported document (PDF, PNG, JPEG, TIFF) to the `input/` prefix. This triggers the pipeline; the structured result appears under `output/` within seconds. + +```bash +BUCKET= + +# 1. Upload a document to the input/ prefix (triggers the pipeline) +aws s3 cp ./my-document.pdf s3://$BUCKET/input/my-document.pdf + +# 2. Wait ~20-30 seconds, then list the output +aws s3 ls s3://$BUCKET/output/ --recursive + +# 3. Read the structured result (replace with the folder name from step 2) +aws s3 cp s3://$BUCKET/output//0/standard_output/0/result.json - +``` + +The `result.json` contains the extracted document text as Markdown (including tables) and a generative summary under the `document` and `pages` keys. + +## Changing what is extracted + +Edit the `BDAProject` resource in `template.yaml`: +- Add `Image`, `Video`, or `Audio` blocks to `StandardOutputConfiguration` to process other media types. +- Attach a `CustomOutputConfiguration` (a blueprint) to extract a specific set of fields (for example invoice number, total, and line items) as typed JSON. + +Note: the system profile `us.data-automation-v1` is a cross-region (US geo) profile, so BDA may route the job to any US region. The IAM policy region-wildcards the profile ARN for this reason. + +## Cleanup + +```bash +aws s3 rm s3://$BUCKET --recursive +sam delete +``` + +---- + +Author: Manish S + +Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0 diff --git a/s3-bedrock-data-automation-sam/example-pattern.json b/s3-bedrock-data-automation-sam/example-pattern.json new file mode 100644 index 000000000..2f406192d --- /dev/null +++ b/s3-bedrock-data-automation-sam/example-pattern.json @@ -0,0 +1,64 @@ +{ + "title": "Extract structured data from documents with Amazon Bedrock Data Automation", + "description": "Automatically turn files uploaded to Amazon S3 (documents, images, video, audio) into structured JSON using Amazon Bedrock Data Automation and AWS Lambda, with no machine-learning code.", + "language": "Python", + "level": "200", + "framework": "AWS SAM", + "patternArch": { + "icon1": { "x": 20, "y": 50, "service": "s3", "label": "Amazon S3 (input + output)" }, + "icon2": { "x": 50, "y": 50, "service": "lambda", "label": "AWS Lambda (starts BDA job)" }, + "icon3": { "x": 80, "y": 50, "service": "bedrock", "label": "Bedrock Data Automation" }, + "line1": { "from": "icon1", "to": "icon2", "label": "ObjectCreated" }, + "line2": { "from": "icon2", "to": "icon3", "label": "invoke async" }, + "line3": { "from": "icon3", "to": "icon1", "label": "structured JSON" } + }, + "introBox": { + "headline": "How it works", + "text": [ + "Amazon Bedrock Data Automation (BDA) is a managed service that extracts structured information from unstructured content such as documents, images, video, and audio. This pattern wires it into a fully serverless, event-driven pipeline so that any file dropped into Amazon S3 is automatically turned into structured JSON, with no machine-learning code to write or maintain.", + "When a file is uploaded to the input/ prefix of the Amazon S3 bucket, the ObjectCreated event triggers an AWS Lambda function. The function calls the BDA runtime InvokeDataAutomationAsync API, passing the input file location, an output location, and the Amazon Bedrock Data Automation project that defines what to extract. The function is fire-and-forget: it only starts the asynchronous job and returns immediately, so it stays small and cheap.", + "Bedrock Data Automation reads the file, runs the managed extraction defined by the project, and writes the structured result to the output/ prefix of the same bucket. The project in this pattern uses a standard output configuration that returns each document as clean Markdown text (including tables) plus a generative summary. You can change what is extracted by editing the project resource in the template: add Image, Video, or Audio blocks, or attach a custom output configuration (a blueprint) that defines the exact fields you want.", + "The Amazon Bedrock Data Automation project is a native AWS CloudFormation resource (AWS::Bedrock::DataAutomationProject), so the entire pipeline is defined as infrastructure as code and the amount of custom code is minimal. IAM permissions follow least privilege: the Lambda function may read and write only the pattern bucket, and may invoke only Bedrock Data Automation. Note that the system data automation profile (us.data-automation-v1) is a cross-region profile, so the profile ARN in the policy is region-wildcarded because the job can be routed to any US region.", + "Good use cases include invoice and form processing, contract and document understanding, media cataloging, and any workflow that needs to convert a pile of raw files into queryable data. Because both Amazon S3 and AWS Lambda are pay-per-use and Bedrock Data Automation is fully managed, the pipeline scales to zero and has no idle cost." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/s3-bedrock-data-automation-sam", + "templateURL": "serverless-patterns/s3-bedrock-data-automation-sam", + "projectFolder": "s3-bedrock-data-automation-sam", + "templateFile": "template.yaml" + } + }, + "resources": { + "headline": "Additional resources", + "bullets": [ + { "text": "Amazon Bedrock Data Automation - documentation", "link": "https://docs.aws.amazon.com/bedrock/latest/userguide/bda.html" }, + { "text": "AWS::Bedrock::DataAutomationProject - CloudFormation resource", "link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-bedrock-dataautomationproject.html" }, + { "text": "InvokeDataAutomationAsync - BDA runtime API", "link": "https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeDataAutomationAsync.html" }, + { "text": "Using Amazon S3 notifications to trigger AWS Lambda", "link": "https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html" } + ] + }, + "deploy": { + "text": [ + "sam build", + "sam deploy --guided" + ] + }, + "testing": { + "headline": "Testing", + "text": [ + "Upload a document to the input/ prefix, then read the structured JSON from the output/ prefix. See the README for detailed instructions." + ] + }, + "cleanup": { + "headline": "Cleanup", + "text": [ + "1. Empty the S3 bucket: aws s3 rm s3://YOUR_BUCKET --recursive.", + "2. Delete the stack: sam delete." + ] + }, + "authors": [ + { "name": "Manish S", "image": "", "bio": "", "linkedin": "", "twitter": "" } + ] +} diff --git a/s3-bedrock-data-automation-sam/src/handler.py b/s3-bedrock-data-automation-sam/src/handler.py new file mode 100644 index 000000000..d44d12629 --- /dev/null +++ b/s3-bedrock-data-automation-sam/src/handler.py @@ -0,0 +1,36 @@ +"""Start an Amazon Bedrock Data Automation (BDA) job for each file uploaded to S3. + +Triggered by an S3 ObjectCreated event on the input/ prefix. Calls the BDA runtime +invoke_data_automation_async API, which reads the uploaded file, runs the managed +extraction defined by the BDA project, and writes structured JSON to the output/ prefix. +The heavy lifting is done by BDA - this function only starts the job. +""" +import os +import urllib.parse +import boto3 + +bda = boto3.client("bedrock-data-automation-runtime") + +PROJECT_ARN = os.environ["BDA_PROJECT_ARN"] +PROFILE_ARN = os.environ["BDA_PROFILE_ARN"] +OUTPUT_PREFIX = os.environ.get("OUTPUT_PREFIX", "output") + + +def handler(event, context): + started = [] + for record in event.get("Records", []): + bucket = record["s3"]["bucket"]["name"] + key = urllib.parse.unquote_plus(record["s3"]["object"]["key"]) + response = bda.invoke_data_automation_async( + inputConfiguration={"s3Uri": "s3://" + bucket + "/" + key}, + outputConfiguration={"s3Uri": "s3://" + bucket + "/" + OUTPUT_PREFIX}, + dataAutomationConfiguration={ + "dataAutomationProjectArn": PROJECT_ARN, + "stage": "LIVE", + }, + dataAutomationProfileArn=PROFILE_ARN, + ) + invocation_arn = response["invocationArn"] + print("Started BDA job " + invocation_arn + " for s3://" + bucket + "/" + key) + started.append(invocation_arn) + return {"startedInvocations": started} diff --git a/s3-bedrock-data-automation-sam/src/requirements.txt b/s3-bedrock-data-automation-sam/src/requirements.txt new file mode 100644 index 000000000..73303e49d --- /dev/null +++ b/s3-bedrock-data-automation-sam/src/requirements.txt @@ -0,0 +1 @@ +boto3>=1.37.0 diff --git a/s3-bedrock-data-automation-sam/template.yaml b/s3-bedrock-data-automation-sam/template.yaml new file mode 100644 index 000000000..bbe264a64 --- /dev/null +++ b/s3-bedrock-data-automation-sam/template.yaml @@ -0,0 +1,119 @@ +AWSTemplateFormatVersion: "2010-09-09" +Transform: AWS::Serverless-2016-10-31 +Description: > + Extract structured data from unstructured files (documents, images, video, audio) + using Amazon Bedrock Data Automation (BDA). Upload a file to the input/ prefix of the + S3 bucket; an AWS Lambda function starts an asynchronous BDA job; BDA writes structured + JSON to the output/ prefix. The extraction is fully managed by BDA - no ML code. + (s3-bedrock-data-automation-sam) + +Parameters: + DataBucketName: + Type: String + Description: Globally-unique name for the S3 bucket that holds input files and BDA output. + +Resources: + DataBucket: + Type: AWS::S3::Bucket + Properties: + BucketName: !Ref DataBucketName + + # Bedrock Data Automation project - a native CloudFormation resource. This standard-output + # config returns each document as Markdown text plus a generative summary. Swap in Image/ + # Video/Audio blocks (or a CustomOutputConfiguration blueprint) to change what is extracted. + BDAProject: + Type: AWS::Bedrock::DataAutomationProject + Properties: + ProjectName: !Sub "${AWS::StackName}-project" + ProjectDescription: Standard document extraction (Markdown text + generative summary). + ProjectType: ASYNC + ProjectStage: LIVE + StandardOutputConfiguration: + Document: + Extraction: + Granularity: + Types: + - DOCUMENT + - PAGE + BoundingBox: + State: DISABLED + GenerativeField: + State: ENABLED + OutputFormat: + TextFormat: + Types: + - MARKDOWN + AdditionalFileFormat: + State: DISABLED + + StartBdaFunctionRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Service: lambda.amazonaws.com + Action: sts:AssumeRole + ManagedPolicyArns: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole + Policies: + - PolicyName: BdaAndS3Access + PolicyDocument: + Version: "2012-10-17" + Statement: + - Sid: S3ReadWrite + Effect: Allow + Action: + - s3:GetObject + - s3:PutObject + Resource: !Sub "arn:aws:s3:::${DataBucketName}/*" + - Sid: InvokeBda + Effect: Allow + Action: bedrock:InvokeDataAutomationAsync + Resource: + - !Sub "arn:aws:bedrock:${AWS::Region}:${AWS::AccountId}:data-automation-project/*" + # us.data-automation-v1 is a cross-region (US geo) profile: BDA may route the + # job to any US region, so the profile ARN region must be wildcarded here. + - !Sub "arn:aws:bedrock:*:${AWS::AccountId}:data-automation-profile/us.data-automation-v1" + + StartBdaFunction: + Type: AWS::Serverless::Function + Properties: + FunctionName: !Sub "${AWS::StackName}-start-bda" + CodeUri: src/ + Handler: handler.handler + Runtime: python3.13 + Architectures: + - arm64 + Timeout: 30 + MemorySize: 128 + Role: !GetAtt StartBdaFunctionRole.Arn + Environment: + Variables: + BDA_PROJECT_ARN: !GetAtt BDAProject.ProjectArn + BDA_PROFILE_ARN: !Sub "arn:aws:bedrock:${AWS::Region}:${AWS::AccountId}:data-automation-profile/us.data-automation-v1" + OUTPUT_PREFIX: output + Events: + InputUpload: + Type: S3 + Properties: + Bucket: !Ref DataBucket + Events: s3:ObjectCreated:* + Filter: + S3Key: + Rules: + - Name: prefix + Value: input/ + +Outputs: + DataBucketName: + Description: Upload files to the input/ prefix; read results from the output/ prefix. + Value: !Ref DataBucket + BDAProjectArn: + Description: ARN of the Bedrock Data Automation project. + Value: !GetAtt BDAProject.ProjectArn + StartBdaFunctionName: + Description: Lambda function that starts BDA jobs on upload. + Value: !Ref StartBdaFunction