Automating automation: updating multiple Azure DevOps pipelines using Powershell scripting

Recently, I had to implement a workaround in the Azure DevOps classic release pipelines. This is relatively simple update - I would need to do next:

1. find release in the Azure DevOps web UI and start release definition editor
2. go into first stage/environment
3. add a new instance of the "Azure Powershell" task into the list of the stage tasks
4. configure this task

• set name
• set Azure subscription
• set Powershell script file path
• set script arguments
• tell the task to use "Latest installed version" of the Azure Powershell module
5. repeat steps 3 and 4 process for 2 other stages/environments in the current release definition
   
   
   
6. repeat previous 5 steps for other 10 pipelines

As you can see, I had extremely tedious task on my hand. I would have to update 11 release pipelines, add and configure 3 * 11 = 33 new tasks. My ballpark estimation for the required "physical" effort became next (assuming I already had prepared strings for copy/paste operations): 

11 pipelines * 3 stages * 16 clicks = 528 mouse clicks

11 pipelines * 3 stages * (3 Ctrl-C + 3 Ctrl-V) = 192 keyboard buttons presses

By the time I finished with the second pipeline, I started to understand that I need to automate this process somehow; otherwise I will make mistakes and kill my wrists (plus the whole process is very boring :-). 

The Azure DevOps REST API is an obvious solution to create automated update process. As with any other API's, there is a learning curve to understand how to authenticate, build requests, parse output, how abstracts connects with each other, etc. Fortunately, I found a "shortcut": Powershell module VSTeam created by Donovan Brown. This module is a Powershell wrapper for Azure DevOps API and really made my life easier. TO start with this module, you can find the details of how to install and configure VSTeam here.

Let's go back to my task - I had 9 more release pipelines to update, and I wanted to automate the whole process for consistency (plus had some coding fun). Since all of the affected pipelines had been created from the same coockie-cutter template release, I knew what I had to do exactly:

1. get task #3 from each stage of my étalon pipeline (stages are different by Azure Service Principal used - "Azure Subscription" of the first screenshot)
2. insert the étalon tasks into all other pipelines as step #3 of the corresponding stages
3. skip already updated pipeline :-)

The result of all this effort is a Powershell script below. I added comments to explain how does it work.

P.S. As an alternative solution, I could convert these pipelines into YAML format, but it is a task for the feature.

Azure Application Gateway: HTTP headers rewrite rules for App Service with AAD authentication

As you probably already know, you can use Azure App Service as backend pool for Application Gateway. The general configuration procedure can be found in the Microsoft documentation. This configuration works fine for simple sites, but in case you App Service uses Azure Active Directory (AAD) for authentication and authorization extra steps required to deal with HTTP redirections related to the AAD authentication flow.

The problem

Azure App Services configured with AAD authentication like this

two HTTP redirects happen during login process.

The first redirect happens when App Service sends un-authenticated user to AAD authorize endpoint  to allow user to login and obtain the ID token from AAD. The redirect URL will be like this one:

https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize?
client_id={clieent_id}
&response_type=id_token
&redirect_uri=https%3A%2F%2Fsomeapp.azurewebsites.net%2F.auth%2Flogin%2Faad%2Fcallback
&response_mode=form_post
&scope=openid
&state=12345
&nonce=678910

This URL (also known as callback URL) contains the address where Azure AD will direct user's browser to POST authentication response after successful login. You can find more details about this process here. 

The second redirect happens as response from the HTTP POST to the authentication callback URL when App Service redirects authenticated user to the initially requested app URL.

Below are examples of these redirects extracted from the browser's development tools.
First redirect (browser accesses address waf.dg20.net that resolves into the Application Gateway frontend IP):

Second redirect:

As you can see, even if the browser tried to access our app using an address assigned to the Application Gateway, after login we will end-up sending HTTP requests directly to App Service by-passing Application Gateway. This would defeat the whole purpose of putting the app behind the Application Gateway. In the case when App Service is properly locked down and have enabled static IP restrictions to enforce access only through Application Gateway, user potentially will see 403 HTTP error after logon:

The solution

The Azure documentation describes this issue here and offers solution (HTTP headers rewrite) here. Unfortunately, the prescribed procedure doesn't account for Azure AD authentication process and only offers a method to 'fix' the second redirect. Honestly speaking, this could be considered a "good enough" solution, but it still exposes App Service native address to the client and will work only if client can hit this address directly after AAD logon process.

The solution below will hide backend address from client and will work with locked down App Service. It will rewrite "Location" header in the both redirection 302 responses using two rules in the single Rewrite Set on the Application Gateway. Both rules check and rewrite 'Location' header in the HTTP response.

1. First redirect rewrite - login redirect to AAD

Condition (If): header "Pattern to match"

(.*)(redirect_uri=https%3A%2F%2F).*\.azurewebsites\.net(.*)$

Action (then): set header value

{http_resp_Location_1}{http_resp_Location_2}{var_host}{http_resp_Location_3}

2. Second redirect rewrite - callback from AAD

Condition (If): header "Pattern to match"

(https:\/\/).*\.azurewebsites\.net(.*)$

Action (then): set header value

https://{var_host}{http_resp_Location_2}

This Rewrite Set must be associated with Application Gateway routing rule to be effective. The set can be used with any routing rule that uses Azure App Service with AAD authentication as a backend and it can significantly simplify gateway configuration, especially in the scenario where multiple sites are hosted.