feat: Idempotency helper utility by to-mc · Pull Request #245 · aws-powertools/powertools-lambda-python

to-mc · 2020-12-16T13:20:04Z

Issue #, if available: #218

Description of changes:

Implementing RFC #218. I'll be working together with @igorlg on this.

Todos

Implement caching
Update caching implementation with LRU / max size
Review/replace error handling implementation (especially the pickle bit, which currently causes security baseline to fail the build)
Add unit tests and complete integration tests
Review/remove table creation logic/implementation
Add logging
Add docs
Add payload validation
Improve exception handling for unrecoverable errors from persistence layer

Checklist

Meet tenets criteria
Update tests
Update docs
PR title follows conventional commit semantics

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…ntation instead

codecov-io · 2021-01-15T13:48:23Z

Codecov Report

Merging #245 (d17275d) into develop (fc08062) will decrease coverage by 0.51%.
The diff coverage is 93.90%.

@@             Coverage Diff             @@
##           develop     #245      +/-   ##
===========================================
- Coverage    99.75%   99.23%   -0.52%     
===========================================
  Files           79       85       +6     
  Lines         2859     3138     +279     
  Branches       119      151      +32     
===========================================
+ Hits          2852     3114     +262     
- Misses           5       14       +9     
- Partials         2       10       +8

Impacted Files	Coverage Δ
aws_lambda_powertools/shared/cache_dict.py	`71.42% <71.42%> (ø)`
...da_powertools/utilities/idempotency/idempotency.py	`92.53% <92.53%> (ø)`
...wertools/utilities/idempotency/persistence/base.py	`95.16% <95.16%> (ø)`
...ambda_powertools/utilities/idempotency/__init__.py	`100.00% <100.00%> (ø)`
...bda_powertools/utilities/idempotency/exceptions.py	`100.00% <100.00%> (ø)`
...ools/utilities/idempotency/persistence/dynamodb.py	`100.00% <100.00%> (ø)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc08062...d17275d. Read the comment docs.

… hash functions from hashlib

aws_lambda_powertools/utilities/idempotency/idempotency.py

aws_lambda_powertools/utilities/idempotency/persistence.py

michaelbrewer

Just some comments so far.

…sistence layer modules

aws_lambda_powertools/utilities/idempotency/idempotency.py

aws_lambda_powertools/utilities/idempotency/persistence/base.py

michaelbrewer

@cakepietoast I added some updated comments.

aws_lambda_powertools/shared/cache_dict.py

nmoutschen · 2021-02-19T07:57:13Z

aws_lambda_powertools/utilities/idempotency/idempotency.py

+    **Processes Lambda's event in an idempotent manner**
+        >>> from aws_lambda_powertools.utilities.idempotency import idempotent, DynamoDBPersistenceLayer
+        >>>
+        >>> persistence_store = DynamoDBPersistenceLayer(event_key="body", table_name="idempotency_store")
+        >>>
+        >>> @idempotent(persistence_store=persistence_store)
+        >>> def handler(event, context):
+        >>>     return {"StatusCode": 200}


Question: Should we explain how it'll work under the hood here? This could help clarify how often this will call DynamoDB, what parameters it'll look at to decide if it's an idempotent request or not, etc.

Question: Should we mention how this stores the event into DynamoDB?

I think if we address this it should be in the documentation. Happy to make changes to the docs based on feedback!

aws_lambda_powertools/utilities/idempotency/persistence/base.py

aws_lambda_powertools/utilities/idempotency/idempotency.py

aws_lambda_powertools/utilities/idempotency/persistence/base.py

aws_lambda_powertools/utilities/idempotency/persistence/dynamodb.py

aws_lambda_powertools/utilities/idempotency/persistence/base.py

aws_lambda_powertools/utilities/idempotency/idempotency.py

nmoutschen

LGTM for the doc. Just 2 nitpicks, but these can be changed in the future.

docs/utilities/idempotency.md

…don't throw the wrong error for INPROGRESS

…or large items

aws_lambda_powertools/shared/cache_dict.py

aws_lambda_powertools/utilities/idempotency/persistence/base.py

tests/functional/idempotency/test_idempotency.py

michaelbrewer

Small review changes

Co-authored-by: Michael Brewer <michael.brewer@gyft.com>

feat: initial commit for idempotency utility

d503fb0

to-mc assigned igorlg and to-mc Dec 16, 2020

to-mc marked this pull request as draft December 16, 2020 13:20

to-mc changed the title ~~[DRAFT] feat: initial commit for idempotency utility~~ feat: Idempotency helper utility Dec 16, 2020

to-mc added area/utilities feature New feature or functionality labels Dec 16, 2020

Tom McCarthy added 8 commits December 16, 2020 15:15

fix: ensure region is configured in botocore for tests

e564c63

chore: ignore security warning for md5 usage

c4d19ba

chore: add debug logging

45d384b

feat: add local caching for idempotency lookups

13e5b09

feat: replace simple dict cache with LRU

1efc27d

feat: remove idempotent exception handling

546e879

feat: remove unused logic to create ddb table - will handle in docume…

60fd336

…ntation instead

fix: remove redundant code from table creation logic

ee46124

Tom McCarthy added 3 commits January 18, 2021 21:32

chore: move tests to own dir

acab091

chore: remove redundant code for exception handling

2a364fd

feat: add payload validation logic and functionality to use different…

4f5d52b

… hash functions from hashlib

michaelbrewer reviewed Jan 19, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/idempotency.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Jan 19, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/idempotency.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Jan 19, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/persistence.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Jan 19, 2021

View reviewed changes

Tom McCarthy added 2 commits January 19, 2021 17:20

feat: optimization to reduce number of database calls, reorganize per…

a19d955

…sistence layer modules

chore: type corrections

0ef52f9

michaelbrewer reviewed Jan 21, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/idempotency.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Jan 21, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/idempotency.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Jan 21, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/persistence/base.py Show resolved Hide resolved

michaelbrewer reviewed Jan 21, 2021

View reviewed changes

chore: add more logging statements

d128b0a

github-actions bot added documentation Improvements or additions to documentation and removed area/utilities labels Feb 15, 2021

to-mc added the area/utilities label Feb 15, 2021

nmoutschen reviewed Feb 19, 2021

View reviewed changes

Tom McCarthy added 5 commits February 19, 2021 12:15

feat: add support for decimals in json serializer

9de6e29

chore: Add docstring for LRU cache dict

a4cc61a

chore: Remove unused status constants

978a6bb

chore: Rename method for clarity

1fd8b5a

chore: Correct example in docstring

022739e

github-actions bot removed the area/utilities label Feb 19, 2021

nmoutschen approved these changes Feb 19, 2021

View reviewed changes

docs/utilities/idempotency.md Outdated Show resolved Hide resolved

docs/utilities/idempotency.md Outdated Show resolved Hide resolved

Tom McCarthy added 2 commits February 19, 2021 15:41

fix: make data attribute of data record optional in get_record so we …

8a2d4fe

…don't throw the wrong error for INPROGRESS

docs: clarify behaviour for concurrent executions and DDB behaviour f…

e6f2d98

…or large items

michaelbrewer reviewed Feb 19, 2021

View reviewed changes

aws_lambda_powertools/shared/cache_dict.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Feb 19, 2021

View reviewed changes

aws_lambda_powertools/shared/cache_dict.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Feb 19, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/persistence/base.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Feb 19, 2021

View reviewed changes

aws_lambda_powertools/utilities/idempotency/persistence/base.py Outdated Show resolved Hide resolved

michaelbrewer reviewed Feb 19, 2021

View reviewed changes

tests/functional/idempotency/test_idempotency.py Show resolved Hide resolved

michaelbrewer suggested changes Feb 19, 2021

View reviewed changes

Tom McCarthy and others added 5 commits February 19, 2021 17:09

Update aws_lambda_powertools/shared/cache_dict.py

4aa8145

Co-authored-by: Michael Brewer <michael.brewer@gyft.com>

Update aws_lambda_powertools/shared/cache_dict.py

c54952c

Co-authored-by: Michael Brewer <michael.brewer@gyft.com>

Update aws_lambda_powertools/utilities/idempotency/persistence/base.py

7832e56

Co-authored-by: Michael Brewer <michael.brewer@gyft.com>

chore: add test for invalid status on data record

fed58fe

Update aws_lambda_powertools/utilities/idempotency/persistence/base.py

b079387

Co-authored-by: Michael Brewer <michael.brewer@gyft.com>

to-mc merged commit 03f7dcd into develop Feb 19, 2021

to-mc deleted the feat/idempotency_helper branch February 19, 2021 16:24

heitorlessa removed the documentation Improvements or additions to documentation label Mar 5, 2021

dependabot bot mentioned this pull request Mar 8, 2021

Bump aws-lambda-powertools from 1.10.5 to 1.11.0 in /lambda/src plus3it/terraform-aws-org-new-account-iam-role#33

Merged

igorlg mentioned this pull request Jul 27, 2021

RFC: Function Idempotency Helper aws-powertools/powertools-lambda#28

Closed

ijemmy mentioned this pull request Aug 17, 2022

RFC: idempotency utility aws-powertools/powertools-lambda-typescript#447

Closed

Conversation

to-mc commented Dec 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes:

Uh oh!

codecov-io commented Jan 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelbrewer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelbrewer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nmoutschen Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

to-mc Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nmoutschen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelbrewer left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

to-mc commented Dec 16, 2020 •

edited

Loading

codecov-io commented Jan 15, 2021 •

edited

Loading