diff options
Diffstat (limited to 'third_party/rust/jsparagus/.metrics')
9 files changed, 320 insertions, 0 deletions
diff --git a/third_party/rust/jsparagus/.metrics/README.md b/third_party/rust/jsparagus/.metrics/README.md new file mode 100644 index 0000000000..f665dbd9c3 --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/README.md @@ -0,0 +1,79 @@ +[![Rust][Rust Badge]][Rust CI Link] +[![NotImplemented Counter][NotImplemented Badge]][NotImplemented Search] +[![Fuzzbug days since][Fuzzbug Days Badge]][Fuzzbugs] +[![Fuzzbug open][Fuzzbug Open Badge]][Open Fuzzbugs] + +# Metrics + +This is the metrics directory. It follows the evolution of the repository separately from the +repostory. You can find the actual metrics in the +[`ci-results`](https://github.com/mozilla-spidermonkey/jsparagus/tree/ci-results) branch of the jsparagus project. This branch is automatically generated using the `create-ci-branch.sh` script found in this directory. If there are issues with your fork, you can remove the `ci-results` branch, and the ci will automatically rerun the `create-ci-branch` script to reset it. Do not push manula data to this repository, it will be lost. + +If you find that the `ci-results` branch has disappeared or been corrupted somehow, you can reset it by deleting it and recreating it. + +``` +git branch -D ci-results +cd .metrics +./create-ci-branch.sh +``` + +The `create-ci-branch.sh` file creates the branch, prepares it, and populates it with data from the past. + +## Making your own metrics +Make sure you do not use data that can not be automatically recovered. We cannot rely on the `ci-results` branch always being present, therefore anything that you write must be recoverable on its own, either by relying on external APIs or through some other mechanism. + +Please update this README if you make any changes. + +## Types of CI Actions +These actions are all found in the `.github/workflows` directory + +1) `Rust.yml` - Run on Pull Request +* runs every time there is a push to master, use for any metrics that are development related. Examples include linting, testing, etc. +2) `ci-push.yml` - Run on Push to `master` +* runs on self contained metrics. An example is the number of `NotImplemented` errors in the codebase. This does not depend on anything external +3) `ci-daily.yml` - Run Daily +* a cron task that runs daily. Useful for metrics that need daily updates +4) `ci-issue.yml` - Run on issue open +* runs each time an issue is opened. Good for tracking types of issues. + + +## Types of data + +These are the types of data that this metrics folder tracks. + +1) Rust Passing + * Ensures our internal tests are passing + * Updates on every pull request to master. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/rust.yml) + +2) NotImplemented Count + * counts number of NotImplemented errors in the codebase. This should slowly rundown to zero + * Updates on every push to master. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-push.yml) + +3) Days Since last Fuzzbug + * tracks the last fuzzbug we saw, if it does not exist, return ∞, otherwise return the last date regardless of state. + * Updates daily, regardless of push. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml) + +4) Fuzzbug open count + * tracks the number of open fuzzbugs + * Updates on issue open. See [this action](https://github.com/mozilla-spidermonkey/jsparagus/.github/workflows/ci-issues.yml) + +5) Percentage of tests passing with SmooshMonkey + * TODO: tracks the number of tests passing without fallback. We should use the try api for this. + * Updates daily, regardless of push. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml) + +6) Percentage of JS compilable with SmooshMonkey + * TODO: see comment about writing bytes to a file in [this repo](https://github.com/nbp/seqrec) + * implementation is dependant on how we get the data. We need a robust solution for importing this data. + +[Rust Badge]: https://github.com/mozilla-spidermonkey/jsparagus/workflows/Rust/badge.svg +[Rust CI Link]: https://github.com/mozilla-spidermonkey/jsparagus/actions?query=branch%3Amaster +[NotImplemented Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fnot-implemented.json +[NotImplemented Search]: https://github.com/mozilla-spidermonkey/jsparagus/search?q=notimplemented&unscoped_q=notimplemented +[Fuzzbug days Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fsince-last-fuzzbug.json +[Fuzzbug Open Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fopen-fuzzbug.json +[Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/issues?utf8=%E2%9C%93&q=label%3AlibFuzzer+ +[Open Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/labels/libFuzzer diff --git a/third_party/rust/jsparagus/.metrics/create-ci-branch.sh b/third_party/rust/jsparagus/.metrics/create-ci-branch.sh new file mode 100755 index 0000000000..2415ed7ce9 --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/create-ci-branch.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +set -ue # its like javascript, everything is allowed unless you prevent it. +shopt -s extglob + +# export the ci_branch we will be using in all shell scripts +export ci_branch=ci_results + +topdir=$(git rev-parse --show-toplevel) + +cd $topdir + +if [ `git branch --list $ci_branch` ] +then + echo "Branch exists" #We don't need to do anything +else + git checkout -b $ci_branch + + # clear out the repostory + git rm -r !(.metrics|.git|tmp) + git rm -r .github + + cp .metrics/generated_README.md README.md + mkdir .metrics/badges + mkdir .metrics/count + + git add . + git commit -m"Initial commit for results branch" + + # scripts needed to populated. Should be self contained with cleanup of extra files + cd .metrics && ./populate_not_implemented.sh + cd $topdir + cd .metrics && ./populate_fuzzbug.sh + + cd $topdir + git add . + git commit -m"Inital run of Populate scripts" +fi diff --git a/third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py b/third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py new file mode 100644 index 0000000000..654908ff1f --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py @@ -0,0 +1,22 @@ +#!/usr/bin/python +import json + +read_filename = 'count/fuzzbug.json' +write_count = 'badges/open-fuzzbug.json' + +open_fuzzbugs = 0 +with open(read_filename, 'r') as f: + filedata = json.load(f) + # the open fuzzbug count. Can be deleted + open_fuzzbugs = len([x for x in filedata if x['closed_at'] is None]) + +# Write fuzzbug count +data = { + "schemaVersion": 1, + "label": "Open FuzzBugs", + "message": str(open_fuzzbugs) if open_fuzzbugs > 0 else "None", + "color": "green" if open_fuzzbugs == 0 else "yellow", +} + +with open(write_count, 'w') as f: + json.dump(data, f, indent=4) diff --git a/third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py b/third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py new file mode 100644 index 0000000000..a2d2b88c09 --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py @@ -0,0 +1,40 @@ +#!/usr/bin/python +import json +from datetime import datetime + +read_filename = 'count/fuzzbug.json' +write_since = 'badges/since-last-fuzzbug.json' + +days_since = None +with open(read_filename, 'r') as f: + filedata = json.load(f) + count = len(filedata) + # the last time we saw a fuzzbug regardless of status + if count > 0: + dt_format = "%Y-%m-%dT%H:%M:%SZ" + fuzzbug_opened = filedata[0]["created_at"] + fuzzbug_date = datetime.strptime(fuzzbug_opened, dt_format) + today = datetime.today() + days_since = (today - fuzzbug_date).days + + +# Write days since last fuzzbug + +def get_color(days): + if days_since is None or days_since > 100: + return "green" + elif days_since > 10: + return "yellow" + else: + return "red" + + +data = { + "schemaVersion": 1, + "label": "Days since last FuzzBug", + "message": str(days_since) if days_since is not None else "Forever", + "color": get_color(days_since), +} + +with open(write_since, 'w') as f: + json.dump(data, f, indent=4) diff --git a/third_party/rust/jsparagus/.metrics/generated_README.md b/third_party/rust/jsparagus/.metrics/generated_README.md new file mode 100644 index 0000000000..af4208a90f --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/generated_README.md @@ -0,0 +1,48 @@ +[![Rust][Rust Badge]][Rust CI Link] +[![NotImplemented Counter][NotImplemented Badge]][NotImplemented Search] +[![Fuzzbug days since][Fuzzbug Days Badge]][Fuzzbugs] +[![Fuzzbug open][Fuzzbug Open Badge]][Open Fuzzbugs] + +# Metrics + +Unlike other branches in this project, this branch is for collecting metrics from the CI. you will +find these files in the `.results` folder. If this branch gets deleted, don't worry. This branch can be auto-generated from the `.metrics` +folder in the main repository. + +## Types of data + +These are the types of data that this metrics folder tracks. + +1) NotImplemented Count + * counts number of NotImplemented errors in the codebase. This should slowly rundown to zero + * Updates on every push to master. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-push.yml) + +2) Days Since last Fuzzbug + * tracks the last fuzzbug we saw, if it does not exist, return ∞, otherwise return the last date regardless of state. + * Updates daily, regardless of push. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml) + +3) Fuzzbug open count + * tracks the number of open fuzzbugs + * Updates daily, regardless of push. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-issues.yml) + +4) Percentage of tests passing with SmooshMonkey + * TODO: tracks the number of tests passing without fallback. We should use the try api for this. + * Updates daily, regardless of push. See [this + action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml) + + +5) Percentage of JS compilable with SmooshMonkey + * TODO: see comment about writing bytes to a file in [this repo](https://github.com/nbp/seqrec) + * implementation is dependant on how we get the data. We need a robust solution for importing this data. + +[Rust Badge]: https://github.com/mozilla-spidermonkey/jsparagus/workflows/Rust/badge.svg +[Rust CI Link]: https://github.com/mozilla-spidermonkey/jsparagus/actions?query=branch%3Amaster +[NotImplemented Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fnot-implemented.json +[NotImplemented Search]: https://github.com/mozilla-spidermonkey/jsparagus/search?q=notimplemented&unscoped_q=notimplemented +[Fuzzbug days Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fsince-last-fuzzbug.json +[Fuzzbug Open Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fopen-fuzzbug.json +[Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/issues?utf8=%E2%9C%93&q=label%3AlibFuzzer+ +[Open Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/labels/libFuzzer diff --git a/third_party/rust/jsparagus/.metrics/not_implemented_badge.py b/third_party/rust/jsparagus/.metrics/not_implemented_badge.py new file mode 100644 index 0000000000..bc522f1fe3 --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/not_implemented_badge.py @@ -0,0 +1,14 @@ +#!/usr/bin/python +import json +import os.path + +filename = 'badges/not-implemented.json' +total_count = os.environ['total_count'] +data = { + "schemaVersion": 1, + "label": "NotImplemented", + "message": total_count, + "color": "green" if total_count == "0" else "yellow", +} +with open(filename, 'w') as f: + json.dump(data, f, indent=4) diff --git a/third_party/rust/jsparagus/.metrics/not_implemented_count.py b/third_party/rust/jsparagus/.metrics/not_implemented_count.py new file mode 100644 index 0000000000..947044eca9 --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/not_implemented_count.py @@ -0,0 +1,19 @@ +#!/usr/bin/python +import json +import os.path + +filename = 'count/not-implemented.json' +if not os.path.isfile(filename): + with open(filename, 'w') as f: + json.dump([], f, indent=4) # initialize with an empty list + +with open(filename, 'r+') as f: + data = json.load(f) + if len(data) == 0 or data[-1]["commit"] != os.environ['current_commit']: + data.append({ + "commit": os.environ['current_commit'], + "total_count": os.environ['total_count'] + }) + f.seek(0) + json.dump(data, f, indent=4) + f.truncate() diff --git a/third_party/rust/jsparagus/.metrics/populate_fuzzbug.sh b/third_party/rust/jsparagus/.metrics/populate_fuzzbug.sh new file mode 100755 index 0000000000..af48d566b1 --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/populate_fuzzbug.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash + +set -ue # its like javascript, everything is allowed unless you prevent it. + +topdir=$(git rev-parse --show-toplevel) + +cd $topdir/.metrics + +url="https://api.github.com/repos/mozilla-spidermonkey/jsparagus/issues?labels=libFuzzer&state=all" + +curl $url > count/fuzzbug.json +python fuzzbug_count_badge.py +git add . +git commit -m"Add Fuzzbug date" +python fuzzbug_date_badge.py + +git add . + +git commit -m"Add Fuzzbug count" diff --git a/third_party/rust/jsparagus/.metrics/populate_not_implemented.sh b/third_party/rust/jsparagus/.metrics/populate_not_implemented.sh new file mode 100755 index 0000000000..3a6200133c --- /dev/null +++ b/third_party/rust/jsparagus/.metrics/populate_not_implemented.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash + +set -ue # its like javascript, everything is allowed unless you prevent it. + +topdir=$(git rev-parse --show-toplevel) + +cd $topdir +# setup: persist the scripts between commits +mkdir -p tmp +cp -r .metrics tmp/ +git checkout master +git pull origin master + +# create the log of commits +git log --format=oneline --since=2020-01-01 | tac | awk '{print $1}' > tmp/commit-list +cd tmp/.metrics + +# do stuff with the commits +for commit in $(cat $topdir/tmp/commit-list) +do + git checkout $commit + # python script pulls from env variables, export those + export total_count=$(find $topdir/rust -iname '*.rs' -type f -exec cat {} + | grep -c -E "(Emit|Parse|ScopeBuild)Error::NotImplemented") + export current_commit=$commit + python not_implemented_count.py + python not_implemented_badge.py +done + +cd $topdir +git checkout $ci_branch + +# replace this file stuff with whatever it is you want to do to get it to the right place in the +# repo +mv -f tmp/.metrics/count/not-implemented.json .metrics/count/not-implemented.json +mv -f tmp/.metrics/badges/not-implemented.json .metrics/badges/not-implemented.json + +# Cleanup: Kill the tmp dir +rm -r tmp + +git add . +git commit -m"Add NotImplemented" |