Blazor Server GraphServiceClient not authenticated in wrapper library

I have a Blazor Server application that authenticates users through AAD. I'm using the Microsoft Graph SDK to query user info such as photo, etc., which I can do using DI to get the GraphServiceClient directly within a component in the Blazor Server app.

[Inject] private GraphServiceClient GraphServiceClient { get; set; }

However, for testing purposes, I've created a wrapper around the GraphServiceClient called IGraphService in another library within the same solution. This is where the problem occurs and the GraphServiceClient fails authentication with:

No account or login hint was passed to the AcquireTokenSilent call

My code is set up as follows:

Program.cs

builder.Services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
                .AddMicrosoftIdentityWebApp(builder.Configuration.GetSection("AzureAd"))
                .EnableTokenAcquisitionToCallDownstreamApi(builder.Configuration.GetValue<string>("Graph:Scopes")?.Split(' '))
                .AddMicrosoftGraph(builder.Configuration.GetSection("Graph"))
                .AddInMemoryTokenCaches();

builder.Services.AddScoped<IGraphService, GraphService>();

appsettings.json

"AzureAd": {
    "Instance": "https://login.microsoftonline.com",
    "Domain": "{domain}.onmicrosoft.com",
    "CallbackPath": "/signin-oidc",
    "TenantId": "{tenantId}",
    "ClientId": "{clientId}",
    "ClientSecret": "{clientSecret}"
},
"Graph": {
    "BaseUrl": "https://graph.microsoft.com/v1.0",
    "Scopes": "user.read user.readbasic.all"
},

The part of the implementation of the GraphService I created to wrap the GraphServiceClient that tries to set up the GraphServiceClient:

return new GraphServiceClient(new DelegateAuthenticationProvider((requestMessage) =>
                {
                    string[] scopes = new[] { "user.read", "user.readbasic.all" };
                    ITokenAcquisition tokenService = _contextAccessor.HttpContext.RequestServices.GetRequiredService<ITokenAcquisition>();
                    var token = tokenService.GetAccessTokenForUserAsync(scopes);     <-- FAILS!!!

                    requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token.Result);
                    return Task.FromResult(0);
                }));

Clearing my cookies, this works fine the first time the app runs, but on restarts, it fails with the same message. Injecting the GraphServiceClient directly into a component works fine too, but I'm trying to wrap it so the components are loosely coupled.

Additionally, if I pass the AccessToken from the Blazor Server project to the wrapper library, it works fine too, but I want to avoid that.

What am I missing? Is my setup not right? Thanks for any help or pointers :)

Update 1

To simplify the issue, I have a component on a page and I can successfully request the user's token in that component, every time.

public sealed partial class HeaderBar
    {
        [Inject] private ITokenAcquisition TokenAcquisition { get; set; } = null!;
        [Inject] private IGraphService GraphService { get; set; } = null!;

        protected override async Task OnInitializedAsync()
        {
            try
            {
                // this works every time!
                string token = await TokenAcquisition.GetAccessTokenForUserAsync(new[] { "user.read", "user.readbasic.all" });

                var photo = await GraphService.GetUserPhoto();
                ...
            }
            catch (Exception ex)
            {
                ...
            }
        }
    }

As you can see, I'm not doing anything with the token, but if I remove that line, the same line within the GraphService fails. It will only succeed and get a token if I make the same call within the component first.

I've tried moving the code to the App.razor, but it fails there too.