Skip to content

Conversation

@jshearer
Copy link
Contributor

@jshearer jshearer commented Dec 11, 2024

We ran into this when a customer changed their storage account name. Fragment listings for their journal were returning errors complaining about resolving the hostname of the previous storage account, even through the fragment spec correctly specified new storage account name.


This change is Reviewable

@jshearer jshearer force-pushed the jshearer/fix-azure-fragment-store-change-storage-account branch from 2184783 to 8e3fa9b Compare December 11, 2024 22:35
@jgraettinger
Copy link
Contributor

is this ready for review?

@jshearer
Copy link
Contributor Author

No, I've been having all sorts of trouble getting a local stack working so I haven't managed to test this yet :/

@jshearer jshearer force-pushed the jshearer/fix-azure-fragment-store-change-storage-account branch 2 times, most recently from 9aeef70 to e8ca09f Compare December 17, 2024 20:08
We ran into this when a customer changed their storage account name. Fragment listings for their journal were using the wrong storage client, and returning errors complaining about resolving the hostname of the previous storage account, even through the fragment spec correctly specified new storage account name.
@jshearer jshearer force-pushed the jshearer/fix-azure-fragment-store-change-storage-account branch from e8ca09f to ccbcfa0 Compare December 18, 2024 03:46
@jshearer jshearer marked this pull request as ready for review December 18, 2024 03:56
@jshearer
Copy link
Contributor Author

jshearer commented Dec 18, 2024

@jgraettinger Okay, finally got my local stack un-broken and validated both that I could reproduce the original problem, and that this fixes it.

  • Start out with a storage mapping like so:
    {
        "provider":"AZURE",
        "storage_account_name":"account_a",
        "container_name":"data",
        "prefix":"collection-data/",
        "account_tenant_id":"1234"
    }
    
  • Create a hello world capture with a short flushInterval
  • Fragments start showing up in account_a, and signed URLs work correctly
  • Add a new storage mapping to a different Azure storage account in the same tenant, keeping the old storage mapping around:
    {
        "provider":"AZURE",
        "storage_account_name":"account_b",
        "container_name":"data",
        "prefix":"collection-data/",
        "account_tenant_id":"1234"
    },
    {
        "provider":"AZURE",
        "storage_account_name":"account_a",
        "container_name":"data",
        "prefix":"collection-data/",
        "account_tenant_id":"1234"
    }
    
  • Re-publish the capture task
  • This is what was broken: New fragments are written to the new storage account, but signed URLs for fragments in account_b generate the following error:
    <Error>
        <Code>AuthenticationFailed</Code>
        <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:8dca77fa-f01e-006a-66fe-507415000000 Time:2024-12-18T03:39:16.6155395Z</Message>
        <AuthenticationErrorDetail>Signature did not match. String to sign used was r 2024-12-18T04:39:13Z /blob/josephtesting/data/collection-data//test/test-9/events/0fa82ffc2c80001e/pivot=00/utc_date=2024-12-18/utc_hour=03/00000000000107e6-00000000000149f7-84bb727e5669067a87a19b31add965ff6f155ef0.gz a40cc9be-df7e-4ac1XXXXXXXXXXXXXXXXXX b668bb48-8340-473a-acb7-72103f4f0b09 2024-12-18T03:27:30Z 2024-12-24T03:27:40Z b 2020-10-02 https 2020-02-10 b </AuthenticationErrorDetail>
    </Error>
    
  • Now, if I swap my Gazette version for the one in this pull request and repeat the above steps, after adding the second storage mapping and re-publishing, signed URLs generated for fragments written to both storage mappings all work as expected.

Copy link
Contributor

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jgraettinger jgraettinger merged commit 79f35f4 into master Dec 21, 2024
1 check passed
@jgraettinger jgraettinger deleted the jshearer/fix-azure-fragment-store-change-storage-account branch December 21, 2024 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants