Day 43 Airflow(2)

๐Ÿ“‹ย ๊ณต๋ถ€ ๋‚ด์šฉ

Airflow ์‹ค์Šต

context & xcom_pull

Connections & Variables

@task decorator

DAG schedule

Airflow ๊ณผ์ œ

  • ๊ณผ์ œ github

  • airflow ์‹คํ–‰ ๋ฐ ์„ฑ๊ณตํ•œ ๊ฒฐ๊ณผ ํ™”๋ฉด (์ฝ”๋“œ ํ…Œ์ŠคํŠธ ํ•˜๋Š๋ผ ๋งŽ์ด๋„ ์‹คํŒจํ•จ)

    Alt text

  • Data warehouse์— ์—ฐ๊ฒฐํ•˜์—ฌ ์ƒ์„ฑ๋œ table ๋‚ด์šฉ์„ ๋ถˆ๋Ÿฌ์˜จ ๊ฒฐ๊ณผ

    Alt text

๐Ÿ‘€ย CHECK

(์–ด๋ ต๊ฑฐ๋‚˜ ์ƒˆ๋กญ๊ฒŒ ์•Œ๊ฒŒ ๋œ ๊ฒƒ ๋“ฑ ๋‹ค์‹œ ํ™•์ธํ•  ๊ฒƒ๋“ค)

SSH ์—ฐ๊ฒฐ

in vscode

  • remote ssh ์„ค์น˜
  • f1 > connect to host
    1
    
    ssh -i {path_to_ssh_secret_key}/{key_name} user@ip
    
  • /Users/{user_name}/.ssh/config
    • root ์œ ์ €๋กœ ์ ‘์†ํ•˜๋Š” ๊ฒฝ์šฐ config ํŒŒ์ผ ๋‚ด์šฉ์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.
    1
    2
    3
    4
    
    Host {ip}
        HostName {ip}
        IdentityFile {path_to_ssh_secret_key}/{key_name}
        User root
    

ssh key ์ƒ์„ฑ๋ฒ•

  • root user๋กœ ์ƒ์„ฑํ•œ ํ‚ค
1
ssh-keygen -t rsa -f {key_name} -C root -b 2048

root user ๋กœ๊ทธ์ธ ๋ฐฉ๋ฒ• (feat. Google Cloud platform)

  • ์ฐธ๊ณ  ๋งํฌ ๋‚ด์šฉ ๊ทธ๋Œ€๋กœ ๋”ฐ๋ผํ•˜๊ธฐ
  • ์ ‘์† ์‹œ์—๋Š” ๊ธฐ์กด gcloud ์ปค๋งจ๋“œ์— root@ ๋ถ™์—ฌ์ฃผ๋ฉด ๋จ
1
gcloud compute ssh --project root@your-project-id --zone your-zone your-instance-name

SQL

single quote in string

  • insert into table_name values (‘Seorim’s name’) -> ์—๋Ÿฌ ๋ฐœ์ƒ

  • replace ’ to '’

    1
    2
    3
    
    text = "Seorim's name"
    text = text.replace("'", "''")
    # Seorim's -> Seorim''s
    
  • insert into table_name values (‘Seorim’’s name’) -> ์ž˜ ์‹คํ–‰๋˜๋Š”๊ฑธ ๋ณผ ์ˆ˜ ์žˆ์Œ :>

โ— ๋Š๋‚€ ์ 

Hugo๋กœ ๋งŒ๋“ฆ
Jimmy์˜ Stack ํ…Œ๋งˆ ์‚ฌ์šฉ ์ค‘