Thursday, October 24, 2019

How to use Azure Web App MSI to verify Data Lake access

Sometimes you need to verify that your Azure web (or function) app can access its data using MSI in Azure Data Lake.

  1. Open web app debug console at https://your-az-weapp.scm.azurewebsites.net/DebugConsole/?shell=powershell
  2. Run next PS script (replace data lake name and path to the target file)
$progressPreference = "silentlyContinue"
$req=Invoke-WebRequest -UseBasicParsing -Uri "$($env:MSI_ENDPOINT)?resource=https://datalake.azure.net/&api-version=2017-09-01" -Headers @{"Secret"="$env:MSI_SECRET"} |ConvertFrom-JSON

$headers = @{}
$headers.Add('x-ms-version','2018-03-28')
$headers.Add('x-ms-client-request-id',[guid]::NewGuid())
$headers.Add('x-ms-date',(Get-Date).AddHours(1).ToString('ddd, dd MMM yyyy HH:MM:ss G\MT'))
$headers.Add('Authorization',"Bearer $($req.access_token)")


$resp=Invoke-WebRequest -UseBasicParsing -Uri "https://somelake.azuredatalakestore.net/webhdfs/v1/Folder/SubFolder/somefile.json?op=GETFILESTATUS&tooid=True&api-version=2018-09-01" -Method GET -Headers $headers
$resp.StatusCode


If it works (and the app has access), you will see "200" HTTP response code:


PS D:\home> $resp.StatusCode
200
PS D:\home>


Otherwise, it would be an error like this one:


Invoke-WebRequest : {"RemoteException":{"exception":"AccessControlException","message":"GETFILESTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.). [af00739c-f9fb-4bfc-8dfd-655169970161] failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.). [af00739c-f9fb-4bf c-8dfd-655169970161][2019-10-24T08:44:20.5577411-07:00]","javaClassName":"org.a pache.hadoop.security.AccessControlException"}}

Stale braches cleanup in Git repo

As code development moves forward, collaboration and experimentation flourish, developers join and leave the team, the Git repos start to ac...