Day 52 Airflow(7)

πŸ“‹Β κ³΅λΆ€ λ‚΄μš©

Google sheet 연동 μ‹€μŠ΅

  • Google Drive API, Google Sheet API λ‘˜ λ‹€ μ‚¬μš©

  • Sheet λ‚΄μš©μ„ μ €μž₯ν•  ν…Œμ΄λΈ” 생성

1
2
3
4
5
6
create table imsolem1226.spreadsheet_copy_testing(
  col1 int,
  col2 int,
  col3 int,
  col4 int
);
  • ν…Œμ΄λΈ”μ— μ €μž₯된 데이터 확인

Airflow API

  • airflow.cfg
1
2
[api]
auth_backend = airflow.api.aut.backend.basic_auth
  • docker-compose.yaml
1
2
3
# AIRFLOW__ : override airflow.cfg
# API__AUTH_BACKENDS : [Api] μ„Ήμ…˜ λ°‘μ˜ 'AUTH_BACKENDS' ν‚€ 값을 μ„€μ •
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth', 'airflow.api.auth.backend.session'
  • command둜 μ„€μ •κ°’ 확인
1
docker exec -it {airflow schedular container name} aiflow config get-value api auth_backend
  • Web UI둜 API μ‚¬μš©μ„ μœ„ν•œ μƒˆλ‘œμš΄ μœ μ € μΆ”κ°€

  • /health API 호좜

1
curl -X GET --user "{name}:{password}" http://{localhost or address ip}:8081/health

  • /config

    • κΈ°λ³Έ μ„€μ • : API둜 μ ‘κ·Ό λΆˆκ°€

    • μ„€μ •κ°’ : default:False, True, non-sensitive-only

    1
    2
    3
    4
    5
    
    #airflow.cfg
    [webserver]
    expose_config = True
    #docker-composer.yaml
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
    
  • ν™œμ„±ν™” 된 DAG μ°ΎκΈ°

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import requests
from requests.auth import HTTPBasicAuth
import json

url = "http://localhost:8080/api/v1/dags"

dags = requests.get(url, auth=HTTPBasicAuth("airflow", "airflow"))

json_dags = json.loads(dags.text) #dags.json()

for dag in json_dags["dags"]:
    if not dag["is_paused"]:
        print(dag)

πŸ‘€Β CHECK

(μ–΄λ ΅κ±°λ‚˜ μƒˆλ‘­κ²Œ μ•Œκ²Œ 된 것 λ“± λ‹€μ‹œ 확인할 것듀)

❗ λŠλ‚€ 점

Hugo둜 λ§Œλ“¦
Jimmy의 Stack ν…Œλ§ˆ μ‚¬μš© 쀑