Skip to content

Commit

Permalink
Update scripts to use requirements-dev.txt
Browse files Browse the repository at this point in the history
  • Loading branch information
pamelafox committed Jun 13, 2023
1 parent cf4eefe commit 975a06f
Show file tree
Hide file tree
Showing 13 changed files with 164 additions and 115 deletions.
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"forwardPorts": [
5000
],
"postCreateCommand": "",
"postCreateCommand": "python3 -m pip install -r requirements-dev.txt",
"remoteUser": "vscode",
"hostRequirements": {
"memory": "8gb"
Expand Down
34 changes: 22 additions & 12 deletions README_azd.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,34 @@

## Deploying with the Azure Developer CLI

> **IMPORTANT:** In order to deploy and run this example, you'll need an **Azure subscription with access enabled for the Azure OpenAI service**. You can request access [here](https://aka.ms/oaiapply). You can also visit [here](https://azure.microsoft.com/free/cognitive-search/) to get some free Azure credits to get you started.
> **IMPORTANT:** In order to deploy and run this example, you'll need an **Azure subscription with access enabled for the Azure OpenAI service**. You can request access [here](https://aka.ms/oaiapply). You can also visit [here](https://azure.microsoft.com/free/cognitive-search/) to get some free Azure credits to get you started. Your Azure Account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner).
> **AZURE RESOURCE COSTS** by default this sample will create Azure App Service and Azure Cognitive Search resources that have a monthly cost, as well as Form Recognizer resource that has cost per document page. You can switch them to free versions of each of them if you want to avoid this cost by changing the parameters file under the infra folder (though there are some limits to consider; for example, you can have up to 1 free Cognitive Search resource per subscription, and the free Form Recognizer resource only analyzes the first 2 pages of each document.)
### Prerequisites
## Opening the project

If you open this project in GitHub Codespaces or a local Dev Container, these will be available in the environment.
Otherwise, you need to install them locally.
This project has [Dev Container support](https://code.visualstudio.com/docs/devcontainers/containers), so it will be be setup automatically if you open it in Github Codespaces or in local VS Code with the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).

- [Azure Developer CLI](https://aka.ms/azure-dev/install)
- [Python 3+](https://www.python.org/downloads/)
- **Important**: Python and the pip package manager must be in the path in Windows for the setup scripts to work.
- [Node.js](https://nodejs.org/en/download/)
- [Git](https://git-scm.com/downloads)
- [Powershell 7+ (pwsh)](https://github.com/powershell/powershell) - For Windows users only.
- **Important**: Ensure you can run `pwsh.exe` from a PowerShell command. If this fails, you likely need to upgrade PowerShell.
If you're not using one of those options for opening the project, then you'll need to:

1. Install prerequisites:

- [Azure Developer CLI](https://aka.ms/azure-dev/install)
- [Python 3+](https://www.python.org/downloads/)
- **Important**: Python and the pip package manager must be in the path in Windows for the setup scripts to work.
- [Node.js](https://nodejs.org/en/download/)
- [Git](https://git-scm.com/downloads)
- [Powershell 7+ (pwsh)](https://github.com/powershell/powershell) - For Windows users only.
- **Important**: Ensure you can run `pwsh.exe` from a PowerShell command. If this fails, you likely need to upgrade PowerShell.

2. Create a [Python virtual environment](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments) and activate it.

3. Install the Python requirements:

```shell
python3 -m pip install -r requirements-dev.txt
```

>NOTE: Your Azure Account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator) or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner).

### Starting from scratch:

Expand Down
9 changes: 7 additions & 2 deletions azure.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ services:
continueOnError: false
hooks:
preprovision:
windows:
shell: pwsh
run: ./scripts/auth_init.ps1
interactive: true
continueOnError: false
posix:
shell: sh
run: ./scripts/auth_init.sh
Expand All @@ -30,11 +35,11 @@ hooks:
postprovision:
windows:
shell: pwsh
run: ./scripts/prepdocs.ps1
run: ./scripts/auth_update.ps1;./scripts/prepdocs.ps1
interactive: true
continueOnError: false
posix:
shell: sh
run: ./scripts/prepdocs.sh
run: ./scripts/auth_update.sh;./scripts/prepdocs.sh
interactive: true
continueOnError: false
File renamed without changes.
10 changes: 10 additions & 0 deletions scripts/auth_init.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
. ./loadenv.ps1

$pythonCmd = Get-Command python -ErrorAction SilentlyContinue
if (-not $pythonCmd) {
# fallback to python3 if python not found
$pythonCmd = Get-Command python3 -ErrorAction SilentlyContinue
}

Write-Host 'Running auth_update.py'
Start-Process -FilePath ($pythonCmd).Source -ArgumentList "./scripts/prepdocs.py --appid $env:AUTH_APP_ID --uri $env:BACKEND_URI" -Wait -NoNewWindow
123 changes: 83 additions & 40 deletions scripts/auth_init.py
Original file line number Diff line number Diff line change
@@ -1,46 +1,89 @@
import argparse
import subprocess

from azure.identity import AzureDeveloperCliCredential
import urllib3

credential = AzureDeveloperCliCredential()

resp = urllib3.request(
"POST",
"https://graph.microsoft.com/v1.0/applications",
headers={
"Authorization": "Bearer "
+ credential.get_token("https://graph.microsoft.com/.default").token,
},
json={
"displayName": "WebApp",
"signInAudience": "AzureADandPersonalMicrosoftAccount",
"web": {"redirectUris": ["http://localhost:5000/.auth/login/aad/callback"],
"implicitGrantSettings": {
"enableIdTokenIssuance": True
}},
},
timeout=urllib3.Timeout(connect=10, read=10),
)

app_id = resp.json()["id"]
client_id = resp.json()["appId"]

# Add a client secret to the application
# using https://graph.microsoft.com/v1.0/applications/{id}/addPassword
# where {id} is the application ID returned from the previous step
resp = urllib3.request(
"POST",
f"https://graph.microsoft.com/v1.0/applications/{app_id}/addPassword",
headers={
def get_auth_headers(credential):
return {
"Authorization": "Bearer "
+ credential.get_token("https://graph.microsoft.com/.default").token,
},
json={"passwordCredential": {"displayName": "WebAppSecret"}},
timeout=urllib3.Timeout(connect=10, read=10),
)
client_secret = resp.json()["secretText"]

subprocess.run(f"azd env set AUTH_APP_ID {app_id}", shell=True)
subprocess.run(f"azd env set AUTH_CLIENT_ID {client_id}", shell=True)
subprocess.run(f"azd env set AUTH_CLIENT_SECRET {client_secret}", shell=True)
+ credential.get_token("https://graph.microsoft.com/.default").token
}

def check_for_application(credential, app_id):
resp = urllib3.request(
"GET",
f"https://graph.microsoft.com/v1.0/applications/{app_id}",
headers=get_auth_headers(credential))
if resp.status != 200:
print("Application not found")
return False
return True


def create_application(credential):
resp = urllib3.request(
"POST",
"https://graph.microsoft.com/v1.0/applications",
headers=get_auth_headers(credential),
json={
"displayName": "WebApp",
"signInAudience": "AzureADandPersonalMicrosoftAccount",
"web": {"redirectUris": ["http://localhost:5000/.auth/login/aad/callback"],
"implicitGrantSettings": {
"enableIdTokenIssuance": True
}},
},
timeout=urllib3.Timeout(connect=10, read=10),
)

app_id = resp.json()["id"]
client_id = resp.json()["appId"]

return app_id, client_id

def add_client_secret(credential, app_id):
resp = urllib3.request(
"POST",
f"https://graph.microsoft.com/v1.0/applications/{app_id}/addPassword",
headers=get_auth_headers(credential),
json={"passwordCredential": {"displayName": "WebAppSecret"}},
timeout=urllib3.Timeout(connect=10, read=10),
)
client_secret = resp.json()["secretText"]
return client_secret

def update_azd_env(name, val):
subprocess.run(f"azd env set {name} {val}", shell=True)

if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Create an App Registration and client secret (if not already created)",
epilog="Example: auth_update.py",
)
parser.add_argument(
"--appid",
required=False,
help="Optional. ID of registered application. If provided, this script just makes sure it exists.",
)
args = parser.parse_args()

credential = AzureDeveloperCliCredential()

if args.appid:
print(f"Checking if application {args.appid} exists")
if check_for_application(credential, args.appid):
print("Application already exists, not creating new one.")
exit(0)

print("Creating application registration")
app_id, client_id = create_application(credential)

print(f"Adding client secret to {app_id}")
client_secret = add_client_secret(credential, app_id)

print("Updating azd env with AUTH_APP_ID, AUTH_CLIENT_ID, AUTH_CLIENT_SECRET")
update_azd_env("AUTH_APP_ID", app_id)
update_azd_env("AUTH_CLIENT_ID", client_id)
update_azd_env("AUTH_CLIENT_SECRET", client_secret)

19 changes: 2 additions & 17 deletions scripts/auth_init.sh
Original file line number Diff line number Diff line change
@@ -1,21 +1,6 @@
#!/bin/sh

echo ""
echo "Loading azd .env file from current environment"
echo ""

while IFS='=' read -r key value; do
value=$(echo "$value" | sed 's/^"//' | sed 's/"$//')
export "$key=$value"
done <<EOF
$(azd env get-values)
EOF

echo 'Creating python virtual environment "scripts/.venv"'
python3 -m venv scripts/.venv

echo 'Installing dependencies from "requirements.txt" into virtual environment'
./scripts/.venv/bin/python -m pip install -r scripts/requirements.txt
. ./scripts/loadenv.sh

echo 'Running "auth_init.py"'
./scripts/.venv/bin/python ./scripts/auth_init.py
python3 ./scripts/auth_init.py --appid "$AUTH_APP_ID"
16 changes: 9 additions & 7 deletions scripts/auth_update.py
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
import argparse
import subprocess

from azure.identity import AzureDeveloperCliCredential
import urllib3


def update_redirect_uris(credential, app_id, uri):
redirect_uris = [
"http://localhost:5000/.auth/login/aad/callback",
f"{uri}/.auth/login/aad/callback",
]
urllib3.request(
"PATCH",
f"https://graph.microsoft.com/v1.0/applications/{app_id}",
Expand All @@ -15,19 +18,16 @@ def update_redirect_uris(credential, app_id, uri):
},
json={
"web": {
"redirectUris": [
"http://localhost:5000/.auth/login/aad/callback",
f"{uri}/.auth/login/aad/callback",
]
"redirectUris": redirect_uris
}
},
)


if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Prepare documents by extracting content from PDFs, splitting content into sections and indexing in a search index.",
epilog="Example: prepdocs.py --searchservice mysearch --index myindex",
description="Add a redirect URI to a registered application",
epilog="Example: auth_update.py --appid 123 --uri https://abc.azureservices.net",
)
parser.add_argument(
"--appid",
Expand All @@ -42,4 +42,6 @@ def update_redirect_uris(credential, app_id, uri):
args = parser.parse_args()

credential = AzureDeveloperCliCredential()

print(f"Updating application registration {args.appid} with redirect URI for {args.uri}")
update_redirect_uris(credential, args.appid, args.uri)
6 changes: 6 additions & 0 deletions scripts/auth_update.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/sh

. ./scripts/loadenv.sh

echo 'Running "auth_update.py"'
python3 ./scripts/auth_update.py --appid "$AUTH_APP_ID" --uri "$BACKEND_URI"
13 changes: 13 additions & 0 deletions scripts/loadenv.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Write-Host "Loading azd .env file from current environment"

$output = azd env get-values

foreach ($line in $output) {
if (!$line.Contains('=')) {
continue
}

$name, $value = $line.Split("=")
$value = $value -replace '^\"|\"$'
[Environment]::SetEnvironmentVariable($name, $value)
}
8 changes: 8 additions & 0 deletions scripts/loadenv.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
echo "Loading azd .env file from current environment"

while IFS='=' read -r key value; do
value=$(echo "$value" | sed 's/^"//' | sed 's/"$//')
export "$key=$value"
done <<EOF
$(azd env get-values)
EOF
18 changes: 1 addition & 17 deletions scripts/prepdocs.ps1
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
Write-Host ""
Write-Host "Loading azd .env file from current environment"
Write-Host ""

$output = azd env get-values

Expand All @@ -14,26 +12,12 @@ foreach ($line in $output) {
[Environment]::SetEnvironmentVariable($name, $value)
}

Write-Host "Environment variables set."

$pythonCmd = Get-Command python -ErrorAction SilentlyContinue
if (-not $pythonCmd) {
# fallback to python3 if python not found
$pythonCmd = Get-Command python3 -ErrorAction SilentlyContinue
}

Write-Host 'Creating python virtual environment "scripts/.venv"'
Start-Process -FilePath ($pythonCmd).Source -ArgumentList "-m venv ./scripts/.venv" -Wait -NoNewWindow

$venvPythonPath = "./scripts/.venv/scripts/python.exe"
if (Test-Path -Path "/usr") {
# fallback to Linux venv path
$venvPythonPath = "./scripts/.venv/bin/python"
}

Write-Host 'Installing dependencies from "requirements.txt" into virtual environment'
Start-Process -FilePath $venvPythonPath -ArgumentList "-m pip install -r ./scripts/requirements.txt" -Wait -NoNewWindow

Write-Host 'Running "prepdocs.py"'
$cwd = (Get-Location)
Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prepdocs.py --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --tenantid $env:AZURE_TENANT_ID" -Wait -NoNewWindow
Start-Process -FilePath ($pythonCmd).Source -ArgumentList "./scripts/prepdocs.py --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --tenantid $env:AZURE_TENANT_ID" -Wait -NoNewWindow
21 changes: 2 additions & 19 deletions scripts/prepdocs.sh
Original file line number Diff line number Diff line change
@@ -1,23 +1,6 @@
#!/bin/sh

echo ""
echo "Loading azd .env file from current environment"
echo ""
. ./scripts/loadenv.sh

while IFS='=' read -r key value; do
value=$(echo "$value" | sed 's/^"//' | sed 's/"$//')
export "$key=$value"
done <<EOF
$(azd env get-values)
EOF

echo 'Creating python virtual environment "scripts/.venv"'
python3 -m venv scripts/.venv

echo 'Installing dependencies from "requirements.txt" into virtual environment'
./scripts/.venv/bin/python -m pip install -r scripts/requirements.txt

echo 'Running "auth_update.py"'
./scripts/.venv/bin/python ./scripts/auth_update.py --appid "$AUTH_APP_ID" --uri "$BACKEND_URI"
echo 'Running "prepdocs.py"'
./scripts/.venv/bin/python ./scripts/prepdocs.py --searchservice "$AZURE_SEARCH_SERVICE" --index "$AZURE_SEARCH_INDEX" --formrecognizerservice "$AZURE_FORMRECOGNIZER_SERVICE" --tenantid "$AZURE_TENANT_ID"
python3 ./scripts/prepdocs.py --searchservice "$AZURE_SEARCH_SERVICE" --index "$AZURE_SEARCH_INDEX" --formrecognizerservice "$AZURE_FORMRECOGNIZER_SERVICE" --tenantid "$AZURE_TENANT_ID"

0 comments on commit 975a06f

Please sign in to comment.