Skip to content

Intermittent Failure Resolving Dynamic Variables in Serverless Framework > v4.4.3 #13102

Open
@geekymehta

Description

@geekymehta

Issue description

We are facing intermittent issues resolving dynamic variables—especially SSM/SecretsManager-based secrets and stage/environment-dependent values—when using Serverless Framework versions beyond v4.4.3.

These failures occur both during serverless deploy -s dev and serverless offline start -s dev. About 70% of the time, variable resolution fails, while the remaining attempts succeed without any changes to configuration or credentials.

We're now suspecting the issue is not limited to SSM resolution alone, but could be linked to dynamic variable interpolation (e.g., ${self:custom.ssm.${self:provider.stage}.openai-api-key}) or stage/environment usage across files.

Error:

✖ Error: Cannot resolve '${self:custom.ssm.dev.openai-api-key}' variable at 'provider.environment.OPENAI_API_KEY'. No value is available for this variable, and no default value was provided. Please check your variable definitions or provide a default value.
    at _ResolverManager.resolve (file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:55913)
    at async #handlePlaceholderNode (file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:54343)
    at async file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:52335
    at async processNodeAndHandleCompletion (file:///home/user/.serverless/releases/4.4.4/package/dist/sf-core.js:655:43086)

Environment:

  • Node.js: v22.15.0
  • Serverless Framework: Affected on 4.4.4, 4.14.3, 4.17.0, 4.17.1 (works on 4.4.2)
  • AWS CLI v2, using AWS SSO login via aws-sso-credential-getter

Project Configuration Overview:

// serverless.yml:

custom: ${file(serverless_custom.yml)}
provider:
  name: aws
  profile: ${env:my-aws-profile}
  runtime: nodejs20.x
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}
  environment: ${file(serverless_environment.yml)}

// serverless_environment.yml:

OPENAI_API_KEY: ${self:custom.ssm.${self:provider.stage}.openai-api-key}
MOMENTO_API_KEY: ${self:custom.ssm.${self:provider.stage}.momento-api-key}
S3_UPLOAD_BUCKET: spicychat-assets-${self:provider.stage}
FIREHOSE_STREAM_NAME: ${self:custom.firehoseStreamName.${self:provider.stage}}

// serverless_custom.yml:

ssm:
  prod: ${ssm:/aws/reference/secretsmanager/spicychat/${sls:stage}}
  dev: ${ssm:/aws/reference/secretsmanager/spicychat/${sls:stage}}
  uat: ${ssm:/aws/reference/secretsmanager/spicychat/${sls:stage}}

Steps to Reproduce:

  • Authenticate using:
export AWS_PROFILE=<my-aws-profile>
aws sso login --profile <my-aws-profile>
ssocred <my-aws-profile>
  • Run
serverless deploy -s dev 

OR

serverless offline start -s dev
  • Observe that sometimes all environment variables resolve correctly, but most times critical ones like OPENAI_API_KEY, MOMENTO_API_KEY, etc., are resolved as null.

What we've tried

  • Verified all secrets exist in AWS Secrets Manager.
  • Cleared .serverless cache directory.
  • Confirmed valid SSO session and that AWS profile environment variables are set.
  • Reverted to v4.4.2 — the last known stable version where secrets resolve consistently.

Expected Behavior:

Serverless should consistently resolve SSM secrets when valid AWS credentials are provided and referenced correctly in the configuration.

Actual Behavior:

SSM secrets intermittently fail to resolve, leading to deployment and local development failures. Variables resolve to null, and placeholders like ${self:custom.ssm.dev.openai-api-key} trigger fatal errors due to unresolved values.

Debug Logs: s:core: { serverless_framework: '4.4.4' } s:main: Initializing s:core:router: { command: [ 'offline', 'start' ], options: { s: 'dev', debug: '*' }, versions: { serverless_framework: '4.4.4' }, customConfigFilePath: null, composeParams: undefined, composeResolverProviders: undefined, isWithinCompose: false, composeOrgName: null, composeServiceName: null } ......... ......... ......... s:core:resolver:manager: adding resolver provider opt with config { type: 'opt' } s:core:resolver:manager: provider 'opt', resolver 'options' resolved 'region' to 'null' - skipping to next fallback. s:core:resolver:manager: resolved ${opt:region, 'us-east-1'} to us-east-1 s:core:auth:get-authenticated-data: license key provided manually s:core:resolver:manager: adding resolver provider env with config { type: 'env' } s:core:resolver:manager: resolved ${env:my-aws-profile} to s:core:resolver:manager: adding resolver provider file with config { type: 'file' } s:core:resolver:manager: adding resolver provider self with config { type: 'self' } s:core:resolver:manager: resolved ${opt:stage, 'dev'} to dev s:core:resolver:manager: resolved ${opt:stage, 'dev'} to dev s:core:resolver:manager: resolved ${self:service} to spicychat-backend s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${file(serverless_custom.yml)} to [object Object] s:core:resolver:manager: adding resolver for aws credential resolver ssm s:core:resolver:manager: adding resolver provider sls with config { type: 'sls' } s:core:resolver:manager: resolved ${self:custom.cors.allowCredentials.dev, ''} to false s:core:resolver:manager: resolved ${self:custom.cors.allowedOrigins.dev} to * s:core:resolver:manager: resolved ${sls:stage} to dev s:core:resolver:manager: resolved ${sls:stage} to dev s:core:resolver:manager: resolved ${sls:stage} to dev s:core:resolver:default-aws-credential-resolver: resolving AWS credentials using the AWS SDK credential provider chain with profile: s:core:resolver:manager: resolved ${file(src/character/translation/stateMachine.yml)} to [object Object] s:core:resolver:manager: adding resolver provider aws with config { type: 'aws' } s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${file(serverless_environment.yml)} to [object Object] s:core:resolver:manager: adding resolver for aws credential resolver cf-spicychat-backend-${self:provider.stage}.ApiId s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:service} to spicychat-backend s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.region} to us-east-1 s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.region} to us-east-1 s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:service} to spicychat-backend s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: resolved ${self:provider.region} to us-east-1 s:core:resolver:manager: resolved ${self:provider.stage} to dev s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.openai-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.momento-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-public-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-not-public-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.typesense-activity-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.jwt-signing-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.myscale-password' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.CLICKHOUSE_PASSWORD' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.recombee-private-token' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.tapfiliate-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.stickyio-credentials' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.contentful-management-token' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.contentful-api-key' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.axiom-token' to 'null' - skipping to next fallback. s:core:resolver:manager: provider 'self', resolver 'config' resolved 'custom.ssm.dev.SIGHT_ENGINE_API_SECRET' to 'null' - skipping to next fallback. s:core:resolver:manager: resolved ${self:provider.environment.WAITINQUEUE_SQS} to spicychat-waitingqueue-dev s:core:resolver:manager: resolved ${self:custom.firehoseStreamName.dev} to spicychat-stream-v2-dev s:core:resolver:manager: resolved ${self:custom.axiomDataset.spicychat-backend} to nextday-backend s:core:meta: Skipping analysis event because license key is used s:core:meta: saving meta record to global .serverless directory. s:core:resolver:manager: resolved ${file(./serverless-stacks/spicychat-backend/sqs.yml)} to [object Object] s:core:resolver:manager: resolved ${file(./serverless-stacks/spicychat-backend/lambda-functions.yml)} to ${file(src/chat/queue/config.yml)},${file(src/chat/queue/routes.yml)},${file(src/chat/message/routes.yml)},${file(src/chat/conversation/routes.yml)},${file(src/chat/conversation/config.yml)},${file(src/chat/domain/routes.yml)},${file(src/chat/voice/domain/routes.yml)},${file(src/m2m/routes.yml)},${file(src/order/routes.yml)},${file(src/user/persona/routes.yml)},${file(src/user/domain/routes.yml)},${file(src/user/domain/config.yml)},${file(src/user/setting/routes.yml)},${file(src/subscription/config.yml)},${file(src/subscription/routes.yml)},${file(src/shared/system/config.yml)},${file(src/shared/crm/config.yml)},${file(src/shared/content/routes.yml)},${file(src/shared/image/routes.yml)},${file(src/shared/event/routes.yml)},${file(src/shared/growthbook/routes.yaml)},${file(src/character/domain/trigger/config.yml)},${file(src/character/recommendation/config.yml)},${file(src/character/domain/routes.yml)},${file(src/character/translation/routes.yml)} s:core:resolver:manager: resolved ${file(./serverless-stacks/spicychat-backend/dynamodb-tables.yml)} to [object Object] s:core:resolver:manager: resolved ${aws:region} to us-east-1 s:cli-error-handler: Error: Cannot resolve '${self:custom.ssm.dev.openai-api-key}' variable at 'provider.environment.OPENAI_API_KEY'. No value is available for this variable, and no default value was provided. Please check your variable definitions or provide a default value. at _ResolverManager.resolve (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:667:11) at async _ResolverManager.#handlePlaceholderNode (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:584:9) at async (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:477:11) at async processNodeAndHandleCompletion (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/graph.js:20:5)

✖ Error: Cannot resolve '${self:custom.ssm.dev.openai-api-key}' variable at 'provider.environment.OPENAI_API_KEY'. No value is available for this variable, and no default value was provided. Please check your variable definitions or provide a default value.
at _ResolverManager.resolve (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:667:11)
at async _ResolverManager.#handlePlaceholderNode (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:584:9)
at async (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/manager.js:477:11)
at async processNodeAndHandleCompletion (/home/user/.serverless/releases/4.4.4/sf-core/src/lib/resolvers/graph.js:20:5)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      pFad - Phonifier reborn

      Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

      Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


      Alternative Proxies:

      Alternative Proxy

      pFad Proxy

      pFad v3 Proxy

      pFad v4 Proxy