年轻人的第一次CI——使用GitHub Actions同步北洋园PT静态文件以便jsDelivr分发

杯杯杯杯具 2020-09-25 PM 94℃ 2条

file_6124678.png

其实写代码已经是7月中旬的事了,然而博客却咕到了9月底(逃

由于安全和速度考虑,北洋园PT使用了Cloudflare作为CDN,但是Cloudflare在今年以来就一直被污染、屏蔽,在某些地区成了名副其实的「减速器」。在接到xx条issues后,决定将静态文件从主服务器+Cloudflare转至GitHub并使用jsDelivr的国内节点进行加速。然而北洋园的NexusPHP修改版仓库因为日常被滥用,已于去年闭源,而jsDelivr对GitHub Raw的加速必须要Public Repo,因此就需要单独开一个Repo来同步北洋园PT的静态文件。

听起来只是「打开冰箱-把大象放进去-关上冰箱」的事,干起来却并不算很简单。主要在于这几部分:

  1. 将引用的相对路径改为绝对路径
  2. 将CSS中图片引用的相对路径改为绝对路径
  3. 在涉及到JS和CSS的commit被提交后自动更新静态文件版本(而非使用NexusPHP原本的$cssupdatedate
  4. 整理遗落在各处的JS和CSS文件

整理仓库中的JS和CSS文件

其实整理文件已经可以追溯到5月31日的cd6f8cb,当时在八蠢和R酱的提议下整理了一波北洋园的JS文件,例如把ajaxbasic.js直接删去,用jQuery的ajax替代;为common.js添加注释并删去无用的function,优化现有function的逻辑等等。不过当时还没有下定决心上jsDelivr(因为怎么想怎么麻烦)。

后面又陆陆续续的把仓库里能用npm版本替换的外部库都替换掉了,比如jscolor->vanilla-picker,jquery-ui的时间选择器->flatpickr等,算是把仓库中所有不该有的都剔除干净了,只剩下了业务相关的JS文件(common.js, curtain_imageresizer.js, suggest.js等)和实在找不到哪里来的外部库(fadomatic.js, domLib.js, domTT.js, domTT_drag.js)。

清理CSS文件的工作就不再赘述,把之前xxx.css.bak的历史遗留都删掉了,反正有Git。

编写GitHub Actions Workflow

最初的版本

.github/workflows/update-assets.yaml

name: Update tjupt/tjupt-assets

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      - name: Checkout zcqian/tjupt
        uses: actions/[email protected]
        with:
          path: 'tjupt'

      - name: Create patch file
        id: patch
        run: |
          cd ${{ github.workspace }}/tjupt
          git diff HEAD~ HEAD -- '*.js' '*.css' > ${{ runner.temp }}/temp.patch
          test -s ${{ runner.temp }}/temp.patch && echo "::set-output name=status::yes" || echo "::set-output name=status::no"

      - name: Checkout tjupt/tjupt-assets
        if: steps.patch.outputs.status == 'yes'
        uses: actions/[email protected]
        with:
          repository: 'tjupt/tjupt-assets'
          path: 'tjupt-assets'

      - name: Patch to tjupt/tjupt-assets
        if: steps.patch.outputs.status == 'yes'
        run: |
          cd ${{ github.workspace }}/tjupt-assets
          git apply ${{ runner.temp }}/temp.patch

      - name: Set up Python 3.x
        if: steps.patch.outputs.status == 'yes'
        uses: actions/[email protected]
        with:
          python-version: '3.x'
          architecture: 'x64'

      - name: Run script to replace relative path of CSS `url()`
        if: steps.patch.outputs.status == 'yes'
        run: python3 ${{ github.workspace }}/tjupt/.github/scripts/replace_relative_path.py

      - name: Commit files in tjupt/tjupt-assets
        id: assets-commit
        if: steps.patch.outputs.status == 'yes'
        run: |
          cd ${{ github.workspace }}/tjupt-assets
          git config --local user.email "[email protected]"
          git config --local user.name "GitHub Action"
          git commit -a -m "Update with https://github.com/zcqian/tjupt/commit/${{ github.sha }}"
          git diff --name-only HEAD~ HEAD -- '*.js' '*.css' > ${{ github.workspace }}/filenames.txt
          export COMMIT_ID=$(git rev-parse HEAD)

      - name: Push changes to tjupt/tjupt-assets
        if: steps.patch.outputs.status == 'yes'
        uses: ad-m/[email protected]
        with:
          directory: 'tjupt-assets'
          repository: 'tjupt/tjupt-assets'
          github_token: ${{ secrets.GITHUB_TOKEN }}

      - name: Run script to replace commit id in php files
        if: steps.patch.outputs.status == 'yes'
        run: python3 ${{ github.workspace }}/tjupt/.github/scripts/replace_commit_id.py

      - name: Commit files in zcqian/tjupt
        if: steps.patch.outputs.status == 'yes'
        run: |
          cd ${{ github.workspace }}/tjupt
          git config --local user.email "[email protected]"
          git config --local user.name "GitHub Action"
          git commit -a -m "refactor(Repo): 更新CDN版本号" -m "https://github.com/tjupt/tjupt-assets/commit/$COMMIT_ID"

      - name: Push changes to zcqian/tjupt
        if: steps.patch.outputs.status == 'yes'
        uses: ad-m/[email protected]
        with:
          directory: 'tjupt'
          repository: 'zcqian/tjupt'
          github_token: ${{ secrets.GITHUB_TOKEN }}

.github/scripts/replace_relative_path.py

import glob
import os
import re

if __name__ == '__main__':
    base_dir = os.environ.get('GITHUB_WORKSPACE')
    files = glob.glob(base_dir + "/tjupt-assets/**/*.css", recursive=True)
    for file in files:
        try:
            with open(file, 'r') as fp:
                css_file = fp.read()
        except UnicodeDecodeError:
            print(file)
            with open(file, 'r', encoding='gbk') as fp:
                css_file = fp.read()
        path = file.replace(base_dir, "").rsplit("/", 1)[0]
        parent_path = path.rsplit("/", 1)[0] + "/" if '/' in path else ""
        css_file = re.sub(r"url\(\.\./(.*)\)", r"url(//tjupt.org/{}\1)".format(parent_path), css_file)  # url(./xxx.xxx)
        css_file = re.sub(r"url\(\./(.*)\)", r"url(//tjupt.org/{}/\1)".format(path), css_file)  # url(../xxx.xxx)
        css_file = re.sub(r"url\(([a-zA-Z0-9._]*)\)", r"url(//tjupt.org/{}/\1)".format(path), css_file)  # url(xxx.xxx)
        css_file = re.sub(r"url\(/(?!/)(.*)\)", r"url(//tjupt.org/\1)", css_file)  # url(/xxx/xxx.xxx)

        with open(file, 'w') as fp:
            fp.write(css_file)

.github/scripts/replace_commit_id.py

import os
import re

if __name__ == '__main__':
    base_dir = os.environ.get('GITHUB_WORKSPACE') + "/tjupt/"
    commit_id = os.environ.get('COMMIT_ID')

    if not commit_id:
        print("No commit id set in environment variants")
        exit(1)

    with open(base_dir + "../filenames.txt", 'r') as fp:
        edited_files = fp.read().split(os.linesep)
    edited_files = filter(lambda f: f != '' and f.strip() != '', edited_files)
    for edited_file in edited_files:
        [filename, ext] = edited_file.rsplit(".", 1)
        if filename == 'userAutoTips':
            search_files = ['index.php', 'include/functions.php']
        else:
            search_files = ['include/functions.php']

        for file in search_files:
            with open(base_dir + file, 'r') as fp:
                php_file = fp.read()

            # 普通文件
            php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{{40}}/{n}(?:\.min)?\.{e}".format(n=filename, e=ext),
                              "gh/tjupt/[email protected]{}/{}.min.{}".format(commit_id, filename, ext), php_file)

            if 'forumsprites' in filename:
                php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{40}/<\?= get_forum_pic_folder\(\) \?>",
                                  "gh/tjupt/[email protected]{}/<?= get_forum_pic_folder() ?>".format(commit_id), php_file)
            if 'theme' in filename or ('DomTT' in filename and ext == 'css'):
                php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{40}/<\?= \$css_uri \?>",
                                  "gh/tjupt/[email protected]{}/<?= $css_uri ?>".format(commit_id), php_file)
            if 'catsprites' in filename:
                php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{40}/\" \. htmlspecialchars",
                                  "gh/tjupt/[email protected]{}/\" . htmlspecialchars".format(commit_id), php_file)

            with open(base_dir + file, 'w') as fp:
                fp.write(php_file)
  1. 拉取tjupt最新源码(一个坑)
  2. 创建HEAD~(上一次提交)与HEAD(这一次提交)文件名为*.js*.css的git diff,写入temp.patch(又一个大坑,下面说)
  3. 拉取tjupt/tjupt-assets
  4. 将temp.patch patch到tjupt-assets的代码上
  5. 运行replace_relative_path.py,将CSS的url()中的相对路径改为绝对路径
  6. 提交到tjupt-assets,获取commit Hash,Push(也有一个坑)
  7. 运行replace_commit_id.py,将此次改动中修改的静态文件在PHP中的引用版本进行更新
  8. 提交到tjupt,Push

说一下我犯的错误:

  • 拉取代码时(actions/[email protected])没有传入fetch-depth,而默认为1会导致只能获取本次提交的版本,因此在HEAD~上改来改去反复摩擦了好几个commits都没有解决,后来仔细读了一下才发现问题
  • 在我反复横跳时,主仓库的一个css文件已经和tjupt/tjupt-assets的文件不一致了,导致patch出现问题需要人工干预,想到以后可能也会出现Actions运行失败的时候,到时候就又要人工干预,于是把patch改为了从tjupt主仓库将js和css全部复制并覆盖到tjupt-assets仓库,这样就使得tjupt-assets仓库中的静态文件一直都是最新版本
  • 提交到tjupt-assets出现的权限问题,本来以为GitHub Actions可以用每次运行生成的GITHUB_TOKEN来提交到同帐号中有权限的仓库,后来才意识到我有权限不代表bot也有权限,因此生成了一个个人的token,让钱总丢到了仓库的Secrets里,才完成的跨仓库提交

最终的Workflow

经过修改,最终所有静态文件都交给jsDelivr进行minify,并在考虑缓存的基础上对引用combine,具体见jsDelivr的文档。由此完成了北洋园PT主要静态文件的CDN切换,最终的所有代码如下。

.github/workflows/update-assets.yml

name: Update tjupt/tjupt-assets

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - name: Checkout zcqian/tjupt
      uses: actions/[email protected]
      with:
        path: 'tjupt'
        fetch-depth: 2

    - name: Create patch file
      id: patch
      run: |
        cd ${{ github.workspace }}/tjupt
        git diff ${{ github.event.before }} ${{ github.sha }} -- '*.js' '*.css' > ${{ runner.temp }}/temp.patch
        test -s ${{ runner.temp }}/temp.patch && echo "::set-output name=status::yes" || echo "::set-output name=status::no"

    - name: Checkout tjupt/tjupt-assets
      if: steps.patch.outputs.status == 'yes'
      uses: actions/[email protected]
      with:
        repository: 'tjupt/tjupt-assets'
        path: 'tjupt-assets'
        persist-credentials: false
        fetch-depth: 2

    - name: Set up Python 3.x
      if: steps.patch.outputs.status == 'yes'
      uses: actions/[email protected]
      with:
        python-version: '3.x'
        architecture: 'x64'

    - name: Run script to replace relative path of CSS `url()`
      if: steps.patch.outputs.status == 'yes'
      run: python3 ${{ github.workspace }}/tjupt/.github/scripts/update_assets.py

    - name: Commit files in tjupt/tjupt-assets
      id: assets-commit
      if: steps.patch.outputs.status == 'yes'
      run: |
        cd ${{ github.workspace }}/tjupt-assets
        git add .
        git diff --name-only -- '*.js' '*.css' > ${{ github.workspace }}/filenames.txt
        git diff --cached --name-only -- '*.js' '*.css' >> ${{ github.workspace }}/filenames.txt
        git config --local user.email "[email protected]"
        git config --local user.name "GitHub Action"
        git commit -a -m "Update assets" -m "https://github.com/zcqian/tjupt/commit/${{ github.sha }}"
        echo "::set-env name=COMMIT_ID::$(git rev-parse HEAD)"

    - name: Push changes to tjupt/tjupt-assets
      if: steps.patch.outputs.status == 'yes'
      uses: ad-m/[email protected]
      with:
        directory: 'tjupt-assets'
        repository: 'tjupt/tjupt-assets'
        github_token: ${{ secrets.API_TOKEN_GITHUB }}

    - name: Run script to replace commit id in php files
      if: steps.patch.outputs.status == 'yes'
      run: python3 ${{ github.workspace }}/tjupt/.github/scripts/replace_commit_id.py

    - name: Commit files in zcqian/tjupt
      if: steps.patch.outputs.status == 'yes'
      run: |
        cd ${{ github.workspace }}/tjupt
        git config --local user.email "[email protected]"
        git config --local user.name "GitHub Action"
        git commit -a -m "refactor(Repo): 更新静态内容CDN版本" -m "https://github.com/tjupt/tjupt-assets/commit/${{ env.COMMIT_ID }}"

    - name: Push changes to zcqian/tjupt
      if: steps.patch.outputs.status == 'yes'
      uses: ad-m/[email protected]
      with:
        directory: 'tjupt'
        repository: 'zcqian/tjupt'
        github_token: ${{ secrets.GITHUB_TOKEN }}

.github/scripts/update_assets.py

import glob
import os
import re
import shutil


def copy_assets():
    base_dir = os.environ.get('GITHUB_WORKSPACE') + "/tjupt"
    to_dir = os.environ.get('GITHUB_WORKSPACE') + "/tjupt-assets"

    files = glob.glob(base_dir + "/**/*.css", recursive=True)
    files.extend(glob.glob(base_dir + "/**/*.js", recursive=True))

    for file in files:
        to_path = file.replace(base_dir, "").lstrip("/")
        if not os.path.exists(os.path.join(to_dir, os.path.dirname(to_path))):
            os.makedirs(os.path.join(to_dir, os.path.dirname(to_path)))
        shutil.copyfile(file, os.path.join(to_dir, to_path))


def replace_relative_path():
    base_dir = os.environ.get('GITHUB_WORKSPACE') + "/tjupt-assets/"
    files = glob.glob(base_dir + "**/*.css", recursive=True)
    for file in files:
        try:
            with open(file, 'r') as fp:
                css_file = fp.read()
        except UnicodeDecodeError:
            print(file)
            with open(file, 'r', encoding='gbk') as fp:
                css_file = fp.read()
        path = file.replace(base_dir, "").rsplit("/", 1)[0]
        parent_path = path.rsplit("/", 1)[0] + "/" if '/' in path else ""
        css_file = re.sub(r"url\(\.\./(.*)\)", r"url(//tjupt.org/{}\1)".format(parent_path), css_file)  # url(./xxx.xxx)
        css_file = re.sub(r"url\(\./(.*)\)", r"url(//tjupt.org/{}/\1)".format(path), css_file)  # url(../xxx.xxx)
        css_file = re.sub(r"url\(([a-zA-Z0-9._]*)\)", r"url(//tjupt.org/{}/\1)".format(path), css_file)  # url(xxx.xxx)
        css_file = re.sub(r"url\(/(?!/)(.*)\)", r"url(//tjupt.org/\1)", css_file)  # url(/xxx/xxx.xxx)

        with open(file, 'w') as fp:
            fp.write(css_file)


if __name__ == '__main__':
    copy_assets()
    replace_relative_path()

.github/scripts/replace_commit_id.py

import os
import re

if __name__ == '__main__':
    base_dir = os.environ.get('GITHUB_WORKSPACE') + "/tjupt/"
    commit_id = os.environ.get('COMMIT_ID')

    if not commit_id:
        print("No commit id set in environment variants")
        exit(1)

    with open(base_dir + "../filenames.txt", 'r') as fp:
        edited_files = fp.read().split(os.linesep)
    edited_files = filter(lambda f: f != '' and f.strip() != '', edited_files)
    for edited_file in edited_files:
        [filename, ext] = edited_file.rsplit(".", 1)
        if filename == 'userAutoTips':
            search_files = ['index.php', 'include/functions.php']
        else:
            search_files = ['include/functions.php']

        for file in search_files:
            with open(base_dir + file, 'r') as fp:
                php_file = fp.read()

            # 普通文件
            php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{{40}}/{n}(?:\.min)?\.{e}".format(n=filename, e=ext),
                              "gh/tjupt/[email protected]{}/{}.min.{}".format(commit_id, filename, ext), php_file)

            if 'forumsprites' in filename:
                php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{40}/<\?= get_forum_pic_folder\(\) \?>",
                                  "gh/tjupt/[email protected]{}/<?= get_forum_pic_folder() ?>".format(commit_id), php_file)
            if 'theme' in filename or ('DomTT' in filename and ext == 'css'):
                php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{40}/<\?= \$css_uri \?>",
                                  "gh/tjupt/[email protected]{}/<?= $css_uri ?>".format(commit_id), php_file)
            if 'catsprites' in filename:
                php_file = re.sub(r"gh/tjupt/[email protected][0-9a-f]{40}/\" \. htmlspecialchars",
                                  "gh/tjupt/[email protected]{}/\" . htmlspecialchars".format(commit_id), php_file)

            with open(base_dir + file, 'w') as fp:
                fp.write(php_file)
标签: CDN, CI

非特殊说明,本博所有文章均为博主原创。

评论啦~



已有 2 条评论


  1. 八月的蠢
    八月的蠢

    tongyi.fan绝赞更新中!

    回复 2020-09-25 14:15
  2. 德莱文
    德莱文

    踩踩

    回复 2020-09-25 18:13