summaryrefslogtreecommitdiffstats
path: root/third_party/rust/jsparagus/.metrics
diff options
context:
space:
mode:
Diffstat (limited to 'third_party/rust/jsparagus/.metrics')
-rw-r--r--third_party/rust/jsparagus/.metrics/README.md79
-rwxr-xr-xthird_party/rust/jsparagus/.metrics/create-ci-branch.sh38
-rw-r--r--third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py22
-rw-r--r--third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py40
-rw-r--r--third_party/rust/jsparagus/.metrics/generated_README.md48
-rw-r--r--third_party/rust/jsparagus/.metrics/not_implemented_badge.py14
-rw-r--r--third_party/rust/jsparagus/.metrics/not_implemented_count.py19
-rwxr-xr-xthird_party/rust/jsparagus/.metrics/populate_fuzzbug.sh19
-rwxr-xr-xthird_party/rust/jsparagus/.metrics/populate_not_implemented.sh41
9 files changed, 320 insertions, 0 deletions
diff --git a/third_party/rust/jsparagus/.metrics/README.md b/third_party/rust/jsparagus/.metrics/README.md
new file mode 100644
index 0000000000..f665dbd9c3
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/README.md
@@ -0,0 +1,79 @@
+[![Rust][Rust Badge]][Rust CI Link]
+[![NotImplemented Counter][NotImplemented Badge]][NotImplemented Search]
+[![Fuzzbug days since][Fuzzbug Days Badge]][Fuzzbugs]
+[![Fuzzbug open][Fuzzbug Open Badge]][Open Fuzzbugs]
+
+# Metrics
+
+This is the metrics directory. It follows the evolution of the repository separately from the
+repostory. You can find the actual metrics in the
+[`ci-results`](https://github.com/mozilla-spidermonkey/jsparagus/tree/ci-results) branch of the jsparagus project. This branch is automatically generated using the `create-ci-branch.sh` script found in this directory. If there are issues with your fork, you can remove the `ci-results` branch, and the ci will automatically rerun the `create-ci-branch` script to reset it. Do not push manula data to this repository, it will be lost.
+
+If you find that the `ci-results` branch has disappeared or been corrupted somehow, you can reset it by deleting it and recreating it.
+
+```
+git branch -D ci-results
+cd .metrics
+./create-ci-branch.sh
+```
+
+The `create-ci-branch.sh` file creates the branch, prepares it, and populates it with data from the past.
+
+## Making your own metrics
+Make sure you do not use data that can not be automatically recovered. We cannot rely on the `ci-results` branch always being present, therefore anything that you write must be recoverable on its own, either by relying on external APIs or through some other mechanism.
+
+Please update this README if you make any changes.
+
+## Types of CI Actions
+These actions are all found in the `.github/workflows` directory
+
+1) `Rust.yml` - Run on Pull Request
+* runs every time there is a push to master, use for any metrics that are development related. Examples include linting, testing, etc.
+2) `ci-push.yml` - Run on Push to `master`
+* runs on self contained metrics. An example is the number of `NotImplemented` errors in the codebase. This does not depend on anything external
+3) `ci-daily.yml` - Run Daily
+* a cron task that runs daily. Useful for metrics that need daily updates
+4) `ci-issue.yml` - Run on issue open
+* runs each time an issue is opened. Good for tracking types of issues.
+
+
+## Types of data
+
+These are the types of data that this metrics folder tracks.
+
+1) Rust Passing
+ * Ensures our internal tests are passing
+ * Updates on every pull request to master. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/rust.yml)
+
+2) NotImplemented Count
+ * counts number of NotImplemented errors in the codebase. This should slowly rundown to zero
+ * Updates on every push to master. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-push.yml)
+
+3) Days Since last Fuzzbug
+ * tracks the last fuzzbug we saw, if it does not exist, return ∞, otherwise return the last date regardless of state.
+ * Updates daily, regardless of push. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml)
+
+4) Fuzzbug open count
+ * tracks the number of open fuzzbugs
+ * Updates on issue open. See [this action](https://github.com/mozilla-spidermonkey/jsparagus/.github/workflows/ci-issues.yml)
+
+5) Percentage of tests passing with SmooshMonkey
+ * TODO: tracks the number of tests passing without fallback. We should use the try api for this.
+ * Updates daily, regardless of push. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml)
+
+6) Percentage of JS compilable with SmooshMonkey
+ * TODO: see comment about writing bytes to a file in [this repo](https://github.com/nbp/seqrec)
+ * implementation is dependant on how we get the data. We need a robust solution for importing this data.
+
+[Rust Badge]: https://github.com/mozilla-spidermonkey/jsparagus/workflows/Rust/badge.svg
+[Rust CI Link]: https://github.com/mozilla-spidermonkey/jsparagus/actions?query=branch%3Amaster
+[NotImplemented Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fnot-implemented.json
+[NotImplemented Search]: https://github.com/mozilla-spidermonkey/jsparagus/search?q=notimplemented&unscoped_q=notimplemented
+[Fuzzbug days Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fsince-last-fuzzbug.json
+[Fuzzbug Open Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fopen-fuzzbug.json
+[Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/issues?utf8=%E2%9C%93&q=label%3AlibFuzzer+
+[Open Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/labels/libFuzzer
diff --git a/third_party/rust/jsparagus/.metrics/create-ci-branch.sh b/third_party/rust/jsparagus/.metrics/create-ci-branch.sh
new file mode 100755
index 0000000000..2415ed7ce9
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/create-ci-branch.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -ue # its like javascript, everything is allowed unless you prevent it.
+shopt -s extglob
+
+# export the ci_branch we will be using in all shell scripts
+export ci_branch=ci_results
+
+topdir=$(git rev-parse --show-toplevel)
+
+cd $topdir
+
+if [ `git branch --list $ci_branch` ]
+then
+ echo "Branch exists" #We don't need to do anything
+else
+ git checkout -b $ci_branch
+
+ # clear out the repostory
+ git rm -r !(.metrics|.git|tmp)
+ git rm -r .github
+
+ cp .metrics/generated_README.md README.md
+ mkdir .metrics/badges
+ mkdir .metrics/count
+
+ git add .
+ git commit -m"Initial commit for results branch"
+
+ # scripts needed to populated. Should be self contained with cleanup of extra files
+ cd .metrics && ./populate_not_implemented.sh
+ cd $topdir
+ cd .metrics && ./populate_fuzzbug.sh
+
+ cd $topdir
+ git add .
+ git commit -m"Inital run of Populate scripts"
+fi
diff --git a/third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py b/third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py
new file mode 100644
index 0000000000..654908ff1f
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/fuzzbug_count_badge.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python
+import json
+
+read_filename = 'count/fuzzbug.json'
+write_count = 'badges/open-fuzzbug.json'
+
+open_fuzzbugs = 0
+with open(read_filename, 'r') as f:
+ filedata = json.load(f)
+ # the open fuzzbug count. Can be deleted
+ open_fuzzbugs = len([x for x in filedata if x['closed_at'] is None])
+
+# Write fuzzbug count
+data = {
+ "schemaVersion": 1,
+ "label": "Open FuzzBugs",
+ "message": str(open_fuzzbugs) if open_fuzzbugs > 0 else "None",
+ "color": "green" if open_fuzzbugs == 0 else "yellow",
+}
+
+with open(write_count, 'w') as f:
+ json.dump(data, f, indent=4)
diff --git a/third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py b/third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py
new file mode 100644
index 0000000000..a2d2b88c09
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/fuzzbug_date_badge.py
@@ -0,0 +1,40 @@
+#!/usr/bin/python
+import json
+from datetime import datetime
+
+read_filename = 'count/fuzzbug.json'
+write_since = 'badges/since-last-fuzzbug.json'
+
+days_since = None
+with open(read_filename, 'r') as f:
+ filedata = json.load(f)
+ count = len(filedata)
+ # the last time we saw a fuzzbug regardless of status
+ if count > 0:
+ dt_format = "%Y-%m-%dT%H:%M:%SZ"
+ fuzzbug_opened = filedata[0]["created_at"]
+ fuzzbug_date = datetime.strptime(fuzzbug_opened, dt_format)
+ today = datetime.today()
+ days_since = (today - fuzzbug_date).days
+
+
+# Write days since last fuzzbug
+
+def get_color(days):
+ if days_since is None or days_since > 100:
+ return "green"
+ elif days_since > 10:
+ return "yellow"
+ else:
+ return "red"
+
+
+data = {
+ "schemaVersion": 1,
+ "label": "Days since last FuzzBug",
+ "message": str(days_since) if days_since is not None else "Forever",
+ "color": get_color(days_since),
+}
+
+with open(write_since, 'w') as f:
+ json.dump(data, f, indent=4)
diff --git a/third_party/rust/jsparagus/.metrics/generated_README.md b/third_party/rust/jsparagus/.metrics/generated_README.md
new file mode 100644
index 0000000000..af4208a90f
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/generated_README.md
@@ -0,0 +1,48 @@
+[![Rust][Rust Badge]][Rust CI Link]
+[![NotImplemented Counter][NotImplemented Badge]][NotImplemented Search]
+[![Fuzzbug days since][Fuzzbug Days Badge]][Fuzzbugs]
+[![Fuzzbug open][Fuzzbug Open Badge]][Open Fuzzbugs]
+
+# Metrics
+
+Unlike other branches in this project, this branch is for collecting metrics from the CI. you will
+find these files in the `.results` folder. If this branch gets deleted, don't worry. This branch can be auto-generated from the `.metrics`
+folder in the main repository.
+
+## Types of data
+
+These are the types of data that this metrics folder tracks.
+
+1) NotImplemented Count
+ * counts number of NotImplemented errors in the codebase. This should slowly rundown to zero
+ * Updates on every push to master. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-push.yml)
+
+2) Days Since last Fuzzbug
+ * tracks the last fuzzbug we saw, if it does not exist, return ∞, otherwise return the last date regardless of state.
+ * Updates daily, regardless of push. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml)
+
+3) Fuzzbug open count
+ * tracks the number of open fuzzbugs
+ * Updates daily, regardless of push. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-issues.yml)
+
+4) Percentage of tests passing with SmooshMonkey
+ * TODO: tracks the number of tests passing without fallback. We should use the try api for this.
+ * Updates daily, regardless of push. See [this
+ action](https://github.com/mozilla-spidermonkey/jsparagus/tree/master/.github/workflows/ci-daily.yml)
+
+
+5) Percentage of JS compilable with SmooshMonkey
+ * TODO: see comment about writing bytes to a file in [this repo](https://github.com/nbp/seqrec)
+ * implementation is dependant on how we get the data. We need a robust solution for importing this data.
+
+[Rust Badge]: https://github.com/mozilla-spidermonkey/jsparagus/workflows/Rust/badge.svg
+[Rust CI Link]: https://github.com/mozilla-spidermonkey/jsparagus/actions?query=branch%3Amaster
+[NotImplemented Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fnot-implemented.json
+[NotImplemented Search]: https://github.com/mozilla-spidermonkey/jsparagus/search?q=notimplemented&unscoped_q=notimplemented
+[Fuzzbug days Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fsince-last-fuzzbug.json
+[Fuzzbug Open Badge]: https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fmozilla-spidermonkey%2Fjsparagus%2Fci_results%2F.metrics%2Fbadges%2Fopen-fuzzbug.json
+[Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/issues?utf8=%E2%9C%93&q=label%3AlibFuzzer+
+[Open Fuzzbugs]: https://github.com/mozilla-spidermonkey/jsparagus/labels/libFuzzer
diff --git a/third_party/rust/jsparagus/.metrics/not_implemented_badge.py b/third_party/rust/jsparagus/.metrics/not_implemented_badge.py
new file mode 100644
index 0000000000..bc522f1fe3
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/not_implemented_badge.py
@@ -0,0 +1,14 @@
+#!/usr/bin/python
+import json
+import os.path
+
+filename = 'badges/not-implemented.json'
+total_count = os.environ['total_count']
+data = {
+ "schemaVersion": 1,
+ "label": "NotImplemented",
+ "message": total_count,
+ "color": "green" if total_count == "0" else "yellow",
+}
+with open(filename, 'w') as f:
+ json.dump(data, f, indent=4)
diff --git a/third_party/rust/jsparagus/.metrics/not_implemented_count.py b/third_party/rust/jsparagus/.metrics/not_implemented_count.py
new file mode 100644
index 0000000000..947044eca9
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/not_implemented_count.py
@@ -0,0 +1,19 @@
+#!/usr/bin/python
+import json
+import os.path
+
+filename = 'count/not-implemented.json'
+if not os.path.isfile(filename):
+ with open(filename, 'w') as f:
+ json.dump([], f, indent=4) # initialize with an empty list
+
+with open(filename, 'r+') as f:
+ data = json.load(f)
+ if len(data) == 0 or data[-1]["commit"] != os.environ['current_commit']:
+ data.append({
+ "commit": os.environ['current_commit'],
+ "total_count": os.environ['total_count']
+ })
+ f.seek(0)
+ json.dump(data, f, indent=4)
+ f.truncate()
diff --git a/third_party/rust/jsparagus/.metrics/populate_fuzzbug.sh b/third_party/rust/jsparagus/.metrics/populate_fuzzbug.sh
new file mode 100755
index 0000000000..af48d566b1
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/populate_fuzzbug.sh
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+set -ue # its like javascript, everything is allowed unless you prevent it.
+
+topdir=$(git rev-parse --show-toplevel)
+
+cd $topdir/.metrics
+
+url="https://api.github.com/repos/mozilla-spidermonkey/jsparagus/issues?labels=libFuzzer&state=all"
+
+curl $url > count/fuzzbug.json
+python fuzzbug_count_badge.py
+git add .
+git commit -m"Add Fuzzbug date"
+python fuzzbug_date_badge.py
+
+git add .
+
+git commit -m"Add Fuzzbug count"
diff --git a/third_party/rust/jsparagus/.metrics/populate_not_implemented.sh b/third_party/rust/jsparagus/.metrics/populate_not_implemented.sh
new file mode 100755
index 0000000000..3a6200133c
--- /dev/null
+++ b/third_party/rust/jsparagus/.metrics/populate_not_implemented.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+set -ue # its like javascript, everything is allowed unless you prevent it.
+
+topdir=$(git rev-parse --show-toplevel)
+
+cd $topdir
+# setup: persist the scripts between commits
+mkdir -p tmp
+cp -r .metrics tmp/
+git checkout master
+git pull origin master
+
+# create the log of commits
+git log --format=oneline --since=2020-01-01 | tac | awk '{print $1}' > tmp/commit-list
+cd tmp/.metrics
+
+# do stuff with the commits
+for commit in $(cat $topdir/tmp/commit-list)
+do
+ git checkout $commit
+ # python script pulls from env variables, export those
+ export total_count=$(find $topdir/rust -iname '*.rs' -type f -exec cat {} + | grep -c -E "(Emit|Parse|ScopeBuild)Error::NotImplemented")
+ export current_commit=$commit
+ python not_implemented_count.py
+ python not_implemented_badge.py
+done
+
+cd $topdir
+git checkout $ci_branch
+
+# replace this file stuff with whatever it is you want to do to get it to the right place in the
+# repo
+mv -f tmp/.metrics/count/not-implemented.json .metrics/count/not-implemented.json
+mv -f tmp/.metrics/badges/not-implemented.json .metrics/badges/not-implemented.json
+
+# Cleanup: Kill the tmp dir
+rm -r tmp
+
+git add .
+git commit -m"Add NotImplemented"