Resiliency policies in Azure Container Apps

In a distributed system, components may fail, networks may experience delays, or nodes may become unreachable. By designing a system to be resilient, you ensure that it can gracefully handle failures and continue providing services. Azure Container Apps recently released a new feature to effortlessly overcome the outbound dependency request failures. Let’s try it out.

Intro

In November 2023 two kind of resiliency policies have been released “Service discovery resiliency” and “DAPR component resiliency”. This article focuses exclusively on the service discovery resiliency.

Key points about the service discovery resiliency:

  • Resiliency policies are configured on the container apps being called.
  • DAPR operates behind the scene but there are no sidecar tobe configured, it is completely transparent.
  • To use this resiliency feature the caller must discover the callee using the ACA service discovery. This means the target ACA should be called by name using one of the following formats : http://destination-app or http://destination-app.{random}.{location}.azurecontainerapps.io.

Four types of resiliency policies are supported: Timeouts, Retries, Circuit breakers and Connection pools. You have the flexibility to choose either one individually or any combination that suits your preference.

Let’s implement a simple service-to-service communication and apply a retry and a circuit breaker. You can find the source code for this this lab (including containers and infrastructure) on this GitHub repo.

Azure Container Apps resiliency

Retries and Circuit breaker

To explore this scenario I have created two extremely simple containers: ‘aca-generator’ which generates N orders and calls the ‘aca-processor’ sequentially. The ‘aca-processor’ that process the order and write in the console logs the status of the processing.

The aca-processor implements the following dummy logic:

  • if the order amount is less than 60 returns HTTP 200.
  • between 60 and 80 returns HTTP 429 (and randomly 200 when retried).
  • greater than 80 returns HTTP 503.

The resiliency policy is applied by deploying this bicep file. 

  • The retry policy allows a maximum of 5 retries only when the failed response contains a specific header (x-retriable-status-code=true).
  • The circuit breaker policy kicks in after 3 consecutive errors and wait 10 seconds before re-evaluating the circuit.

To test the resiliency configuration, we can invoke the aca-generator service using the swagger ui and examine the result in the aca-processor log stream.

Result

In the following picture (Azure Container Apps Log Stream) we can see the resiliency policy in action.

The case of Kelly Moore:

  • 1 - acaprocessor returns 503
  • 2 - the resiliency policy retries the call additional 2 times and then the circuit breaker kicks in.
  • 3 - there are no other calls for 10 seconds and then the circuit is re-evaluated.

The case of Alford Mosciski:

  • a - acaprocessor returns 429
  • b - the call is retried but it fails a second time
  • c - at the third try the call succeed and a HTTP 200 is returned to the caller.

Azure Container Apps resiliency applied

If we look at the replica details of our aca-processor app, we can notice there are no sidecars even if DAPR is used under the hood.

Conclusion

In conclusion, resilience is vital in distributed systems, and Azure Container Apps’ recent Service Discovery Resiliency feature empowers users to effortlessly manage outbound dependency failures. This transparent and flexible solution ensures application robustness with no changes in the application code. Great feature, ContainerApps team!

 Photo by Rita Seneca on Unsplash