Merge branch 'devel' into v4

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: Bug report
+about: Create a report to help us improve MOTH
+labels: bug
+
+---
+
+### Description
+
+<!-- Description of the issue -->
+
+### Steps to Reproduce
+
+1. <!-- First Step -->
+2. <!-- Second Step -->
+3. <!-- and so on… -->
+
+**Expected behavior:**
+
+<!-- What you expect to happen -->
+
+**Actual behavior:**
+
+<!-- What actually happens -->
+
+**Reproduces how often:**
+
+<!-- What percentage of the time does it reproduce? -->
+
+### Versions
+
+<!-- What version of MOTH are you running? Is this happening in mothd or in moth-devel? Are you running in Docker or natively? Also, please include the OS and what version of the OS you're running. -->
+
+### Additional Information
+
+<!-- Any additional information, configuration or data that might be necessary to reproduce the issue. -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,22 @@
+---
+name: Feature request
+about: Suggest an idea for MOTH
+labels: enhancement
+
+---
+
+## Summary
+
+<!-- One paragraph explanation of the feature. -->
+
+## Motivation
+
+<!-- Why are we doing this? What use cases does it support? What is the expected outcome? -->
+
+## Describe alternatives you've considered
+
+<!-- A clear and concise description of the alternative solutions you've considered. Be sure to explain why Atom's existing customizability isn't suitable for this feature. -->
+
+## Additional context
+
+<!-- Add any other context or screenshots about the feature request here. -->

.github/workflows/docker_build_devel.yml

@@ -0,0 +1,12 @@
+name: moth-devel Docker build
+on: [push]
+
+jobs:
+  build-devel:
+    name: Build moth-devel
+    runs-on: ubuntu-latest
+    steps:
+      - name: Retrieve code
+        uses: actions/checkout@v1
+      - name: Build mothd
+        run: docker build -f Dockerfile.moth-devel .

.github/workflows/docker_build_mothd.yml

@@ -0,0 +1,12 @@
+name: Mothd Docker build
+on: [push]
+
+jobs:
+  build-mothd:
+    name: Build mothd
+    runs-on: ubuntu-latest
+    steps:
+      - name: Retrieve code
+        uses: actions/checkout@v1
+      - name: Build mothd
+        run: docker build -f Dockerfile.moth .

CHANGELOG.md

@@ -28,3 +28,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Security
 
+## [Unreleased]
+### Changed
+ - Endpoints `/points.json`, `/puzzles.json`, and `/messages.html` (optional theme file) combine into `/state`
+ - No more `__devel__` category for dev server: this is now `.config.devel` in the `/state` endpoint
+ - Development server no longer serves a static `/` with links: it now redirects you to a randomly-generated seed URL
+ - Default theme modifications to handle all this
+ - Default theme now automatically "logs you in" with Team ID if it's getting state from the devel server
+
+## [v3.5.1] - 2020-03-16
+### Fixed
+- Support insta-checking for legacy puzzles
+
+## [v3.5.0] - 2020-03-13
+### Changed
+- We are now using SHA256 instead of djb2hash
+### Added
+- URL parameter to points.json to allow returning only the JSON for a single
+  team by its team id (e.g., points.json?id=abc123).
+- A CONTRIBUTING.md to describe expectations when contributing to MOTH
+- Include basic metadata in mothballs
+- add_script_stream convenience function allows easy script addition to puzzle
+- Autobuild Docker images to test buildability
+- Extract and use X-Forwarded-For headers in mothd logging
+- Mothballs can now specify `X-Answer-Pattern` header fields, which allow `*`
+  at the beginning, end, or both, of an answer. This is `X-` because we
+  are hoping to change how this works in the future.
+### Fixed
+- Handle cases where non-legacy puzzles don't have an `author` attribute
+- Handle YAML-formatted file and script lists as expected
+- YAML-formatted example puzzle actually works as expected
+- points.log will now always be sorted chronologically
+
+## [3.4.3] - 2019-11-20
+### Fixed
+- Made top-scoring teams full-width
+
+## [3.4.2] - 2019-11-18
+### Fixed
+- Issue with multiple answers in devel server and YAML-format .moth
+
+## [3.4.1] - 2019-11-17
+### Fixed
+- Scoreboard was double-counting points
+
+## [3.4] - 2019-11-13
+### Added
+- A changelog
+- Support for embedding Python libraries at the category or puzzle level
+- Minimal PWA support to permit caching of currently-unlocked content
+- Embedded graph in scoreboard
+- Optional tracking of participant IDs
+- New `notices.html` file for sending broadcast messages to players
+### Changed
+- Use native JS URL objects instead of wrangling everything by hand

CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing to MOTH
+We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+
+## We Develop with Github
+We use github to host code, to track issues and feature requests, as well as accept pull requests.
+
+## We Use [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow), So All Code Changes Happen Through Pull Requests
+Pull requests are the best way to propose changes to the codebase (we use [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)). We actively welcome your pull requests:
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Update [CHANGELOG.md](CHANGELOG.md)
+7. Issue that pull request!
+
+## We Deploy to a Variety of Architectures
+MOTH is most often deployed using Docker, but we strive to ensure that it can easily be run outside of a Docker environment. Please ensure that and changes will not break or substantially alter Dockerized deployments and that, conversely, changes will not so substantially tie MOTH to Docker or particular Docker deployment that it becomes impractical to run MOTH anywhere but inside of Docker
+
+## Any contributions you make will be under the MIT Software License
+When you submit code changes, your submissions are understood to be under the same [MIT License](http://choosealicense.com/licenses/mit/) that covers the project. Feel free to contact the maintainers if that's a concern.
+
+## Report bugs using Github's [issues](https://github.com/dirtbags/moth/issues)
+We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/dirtbags/moth/issues/new); it's that easy!
+
+## Write bug reports with detail, background, and sample code
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+  - Be specific!
+  - Give sample code if you can.
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
+
+## Use a Consistent Coding Style
+
+### Go
+* Run it through `gofmt`
+
+### Javascript
+* We use Javascript ASI
+
+## References
+This document was adapted from the open-source contribution guidelines from [https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62]

Dockerfile.moth

@@ -7,4 +7,6 @@ RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o /mo
 FROM scratch
 COPY --from=builder /mothd /mothd
 COPY theme /theme
+COPY LICENSE.md /LICENSE
+
 ENTRYPOINT [ "/mothd" ]

Dockerfile.moth-devel

@@ -16,6 +16,8 @@ RUN apk --no-cache add \
 COPY devel /app/
 COPY example-puzzles /puzzles/
 COPY theme /theme/
+COPY LICENSE.md /LICENSE
+COPY VERSION /VERSION
 
 ENTRYPOINT [ "python3", "/app/devel-server.py" ]
 CMD [ "--bind", "0.0.0.0:8080", "--puzzles", "/puzzles", "--theme", "/theme" ]

README.md

@@ -1,6 +1,14 @@
 Dirtbags Monarch Of The Hill Server
 =====================
 
+Master:
+![](https://github.com/dirtbags/moth/workflows/Mothd%20Docker%20build/badge.svg?branch=master)
+![](https://github.com/dirtbags/moth/workflows/moth-devel%20Docker%20build/badge.svg?branch=master)
+
+Devel:
+![](https://github.com/dirtbags/moth/workflows/Mothd%20Docker%20build/badge.svg?branch=devel)
+![](https://github.com/dirtbags/moth/workflows/moth-devel%20Docker%20build/badge.svg?branch=devel)
+
 This is a set of thingies to run our Monarch-Of-The-Hill contest,
 which in the past has been called
 "Tracer FIRE",
@@ -67,7 +75,7 @@ you can copy the example puzzles as a starting point:
 
 Then launch the development server:
 
-    $ python3 tools/devel-server.py
+    $ python3 devel/devel-server.py
 
 Point a web browser at http://localhost:8080/
 and start hacking on things in your `puzzles` directory.
@@ -133,6 +141,18 @@ We sometimes to set `teamids.txt` to a bunch of random 8-digit hex values:
 Remember that team IDs are essentially passwords.
 
 
+Enabling offline/PWA mode
+-------------------
+
+If the file `state/export_manifest` is found, the server will expose the 
+endpoint `/current_manifest.json?id=<teamId>`. This endpoint will return 
+a list of all files, including static theme content and JSON and content 
+for currently-unlocked puzzles. This is used by the native PWA 
+implementation and `Cache` button on the index page to cache all of the 
+content necessary to display currently-open puzzles while offline. 
+Grading will be unavailable while offline. Some puzzles may not function 
+as expected while offline. A valid team ID must be provided.
+
 Mothball Directory
 ==================
 
@@ -149,4 +169,7 @@ If you remove a mothball,
 the category will vanish,
 but points scored in that category won't!
 
+Contributing to MOTH
+==================
 
+Please read [CONTRIBUTING.md](CONTRIBUTING.md)

VERSION

@@ -1 +1 @@
-3.3
+3.5.1

cmd/mothd/award.go

@@ -14,6 +14,24 @@ type Award struct {
 	Points   int
 }
 
+type AwardList []*Award
+
+// Implement sort.Interface on AwardList
+func (awards AwardList) Len() int {
+  return len(awards)
+}
+
+func (awards AwardList) Less(i, j int) bool {
+  return awards[i].When.Before(awards[j].When)
+}
+
+func (awards AwardList) Swap(i, j int) {
+  tmp := awards[i]
+  awards[i] = awards[j]
+  awards[j] = tmp
+}
+
+
 func ParseAward(s string) (*Award, error) {
 	ret := Award{}
 

cmd/mothd/award_test.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"testing"
+	"sort"
 )
 
 func TestAward(t *testing.T) {
@@ -38,3 +39,23 @@ func TestAward(t *testing.T) {
 		t.Error("Not throwing error on bad points")
 	}
 }
+
+func TestAwardList(t *testing.T) {
+  a, _ := ParseAward("1536958399 1a2b3c4d counting 1")
+  b, _ := ParseAward("1536958400 1a2b3c4d counting 1")
+  c, _ := ParseAward("1536958300 1a2b3c4d counting 1")
+  list := AwardList{a, b, c}
+  
+  if sort.IsSorted(list) {
+    t.Error("Unsorted list thinks it's sorted")
+  }
+  
+  sort.Stable(list)
+  if (list[0] != c) || (list[1] != a) || (list[2] != b) {
+    t.Error("Sorting didn't")
+  }
+  
+  if ! sort.IsSorted(list) {
+    t.Error("Sorted list thinks it isn't")
+  }
+}

cmd/mothdv3/handlers.go

@@ -8,6 +8,9 @@ import (
 	"log"
 	"net/http"
 	"os"
+	"path"
+	"path/filepath"
+	"regexp"
 	"strconv"
 	"strings"
 )
@@ -73,7 +76,7 @@ func (ctx *Instance) answerHandler(w http.ResponseWriter, req *http.Request) {
 	pointstr := req.FormValue("points")
 	answer := req.FormValue("answer")
 
-	if ! ctx.ValidTeamId(teamId) {
+	if !ctx.ValidTeamId(teamId) {
 		respond(
 			w, req, JSendFail,
 			"Invalid team ID",
@@ -150,9 +153,14 @@ func (ctx *Instance) puzzlesHandler(w http.ResponseWriter, req *http.Request) {
 }
 
 func (ctx *Instance) pointsHandler(w http.ResponseWriter, req *http.Request) {
+	teamId, ok := req.URL.Query()["id"]
+	pointsLog := ctx.jPointsLog
+	if ok && len(teamId[0]) > 0 {
+		pointsLog = ctx.generatePointsLog(teamId[0])
+	}
 	w.Header().Set("Content-Type", "application/json")
 	w.WriteHeader(http.StatusOK)
-	w.Write(ctx.jPointsLog)
+	w.Write(pointsLog)
 }
 
 func (ctx *Instance) contentHandler(w http.ResponseWriter, req *http.Request) {
@@ -217,6 +225,65 @@ func (ctx *Instance) staticHandler(w http.ResponseWriter, req *http.Request) {
 	http.ServeContent(w, req, path, d.ModTime(), f)
 }
 
+func (ctx *Instance) manifestHandler(w http.ResponseWriter, req *http.Request) {
+	if !ctx.Runtime.export_manifest {
+		http.Error(w, "Endpoint disabled", http.StatusForbidden)
+		return
+	}
+
+	teamId := req.FormValue("id")
+	if _, err := ctx.TeamName(teamId); err != nil {
+		http.Error(w, "Must provide a valid team ID", http.StatusUnauthorized)
+		return
+	}
+
+	if req.Method == http.MethodHead {
+		w.WriteHeader(http.StatusOK)
+		return
+	}
+
+	manifest := make([]string, 0)
+	manifest = append(manifest, "puzzles.json")
+	manifest = append(manifest, "points.json")
+
+	// Pack up the theme files
+	theme_root_re := regexp.MustCompile(fmt.Sprintf("^%s/", ctx.ThemeDir))
+	filepath.Walk(ctx.ThemeDir, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+
+		if !info.IsDir() { // Only package up files
+			localized_path := theme_root_re.ReplaceAllLiteralString(path, "")
+			manifest = append(manifest, localized_path)
+		}
+		return nil
+	})
+
+	// Package up files for currently-unlocked puzzles in categories
+	for category_name, category := range ctx.categories {
+		if _, ok := ctx.MaxPointsUnlocked[category_name]; ok { // Check that the category is actually unlocked. This should never fail, probably
+			for _, file := range category.zf.File {
+				parts := strings.Split(file.Name, "/")
+
+				if parts[0] == "content" { // Only pick up content files, not thing like map.txt
+					for _, puzzlemap := range category.puzzlemap { // Figure out which puzzles are currently unlocked
+						if puzzlemap.Path == parts[1] && puzzlemap.Points <= ctx.MaxPointsUnlocked[category_name] {
+
+							manifest = append(manifest, path.Join("content", category_name, path.Join(parts[1:]...)))
+							break
+						}
+					}
+				}
+			}
+		}
+	}
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(http.StatusOK)
+	manifest_json, _ := json.Marshal(manifest)
+	w.Write(manifest_json)
+}
+
 type FurtiveResponseWriter struct {
 	w          http.ResponseWriter
 	statusCode *int
@@ -242,10 +309,22 @@ func (ctx *Instance) ServeHTTP(wOrig http.ResponseWriter, r *http.Request) {
 		w:          wOrig,
 		statusCode: new(int),
 	}
+
+	clientIP := r.RemoteAddr
+
+	if (ctx.UseXForwarded) {
+		forwardedIP := r.Header.Get("X-Forwarded-For")
+		forwardedIP = strings.Split(forwardedIP, ", ")[0]
+
+		if forwardedIP != "" {
+			clientIP = forwardedIP
+		}
+	}
+
 	ctx.mux.ServeHTTP(w, r)
 	log.Printf(
 		"%s %s %s %d\n",
-		r.RemoteAddr,
+		clientIP,
 		r.Method,
 		r.URL,
 		*w.statusCode,
@@ -259,4 +338,5 @@ func (ctx *Instance) BindHandlers() {
 	ctx.mux.HandleFunc(ctx.Base+"/content/", ctx.contentHandler)
 	ctx.mux.HandleFunc(ctx.Base+"/puzzles.json", ctx.puzzlesHandler)
 	ctx.mux.HandleFunc(ctx.Base+"/points.json", ctx.pointsHandler)
+	ctx.mux.HandleFunc(ctx.Base+"/current_manifest.json", ctx.manifestHandler)
 }

cmd/mothdv3/instance.go

@@ -15,6 +15,10 @@ import (
 	"time"
 )
 
+type RuntimeConfig struct {
+	export_manifest bool
+}
+
 type Instance struct {
 	Base            string
 	MothballDir     string
@@ -22,15 +26,18 @@ type Instance struct {
 	StateDir        string
 	ThemeDir        string
 	AttemptInterval time.Duration
-	Debug           bool
+	UseXForwarded   bool
 
-	categories       map[string]*Zipfs
-	update           chan bool
-	jPuzzleList      []byte
-	jPointsLog       []byte
-	nextAttempt      map[string]time.Time
-	nextAttemptMutex *sync.RWMutex
-	mux             *http.ServeMux
+	Runtime RuntimeConfig
+
+	categories        map[string]*Mothball
+	MaxPointsUnlocked map[string]int
+	update            chan bool
+	jPuzzleList       []byte
+	jPointsLog        []byte
+	nextAttempt       map[string]time.Time
+	nextAttemptMutex  *sync.RWMutex
+	mux               *http.ServeMux
 }
 
 func (ctx *Instance) Initialize() error {
@@ -78,6 +85,7 @@ func (ctx *Instance) MaybeInitialize() {
 	os.Remove(ctx.StatePath("until"))
 	os.Remove(ctx.StatePath("disabled"))
 	os.Remove(ctx.StatePath("points.log"))
+
 	os.RemoveAll(ctx.StatePath("points.tmp"))
 	os.RemoveAll(ctx.StatePath("points.new"))
 	os.RemoveAll(ctx.StatePath("teams"))
@@ -134,26 +142,27 @@ func (ctx *Instance) ThemePath(parts ...string) string {
 
 func (ctx *Instance) TooFast(teamId string) bool {
 	now := time.Now()
-	
+
 	ctx.nextAttemptMutex.RLock()
 	next, _ := ctx.nextAttempt[teamId]
 	ctx.nextAttemptMutex.RUnlock()
-	
+
 	ctx.nextAttemptMutex.Lock()
 	ctx.nextAttempt[teamId] = now.Add(ctx.AttemptInterval)
 	ctx.nextAttemptMutex.Unlock()
-	
+
 	return now.Before(next)
 }
 
-func (ctx *Instance) PointsLog() []*Award {
-	var ret []*Award
+func (ctx *Instance) PointsLog(teamId string) AwardList {
+	awardlist := AwardList{}
 
 	fn := ctx.StatePath("points.log")
+
 	f, err := os.Open(fn)
 	if err != nil {
 		log.Printf("Unable to open %s: %s", fn, err)
-		return ret
+		return awardlist
 	}
 	defer f.Close()
 
@@ -165,10 +174,13 @@ func (ctx *Instance) PointsLog() []*Award {
 			log.Printf("Skipping malformed award line %s: %s", line, err)
 			continue
 		}
-		ret = append(ret, cur)
+		if len(teamId) > 0 && cur.TeamId != teamId {
+			continue
+		}
+		awardlist = append(awardlist, cur)
 	}
 
-	return ret
+	return awardlist
 }
 
 // AwardPoints gives points to teamId in category.
@@ -189,7 +201,7 @@ func (ctx *Instance) AwardPoints(teamId, category string, points int) error {
 		return fmt.Errorf("No registered team with this hash")
 	}
 
-	for _, e := range ctx.PointsLog() {
+	for _, e := range ctx.PointsLog("") {
 		if a.Same(e) {
 			return fmt.Errorf("Points already awarded to this team in this category")
 		}

cmd/mothdv3/maintenance.go

@@ -7,16 +7,12 @@ import (
 	"io/ioutil"
 	"log"
 	"os"
+	"sort"
 	"strconv"
 	"strings"
 	"time"
 )
 
-type PuzzleMap struct {
-	Points int
-	Path   string
-}
-
 func (pm *PuzzleMap) MarshalJSON() ([]byte, error) {
 	if pm == nil {
 		return []byte("null"), nil
@@ -33,7 +29,7 @@ func (pm *PuzzleMap) MarshalJSON() ([]byte, error) {
 
 func (ctx *Instance) generatePuzzleList() {
 	maxByCategory := map[string]int{}
-	for _, a := range ctx.PointsLog() {
+	for _, a := range ctx.PointsLog("") {
 		if a.Points > maxByCategory[a.Category] {
 			maxByCategory[a.Category] = a.Points
 		}
@@ -41,45 +37,29 @@ func (ctx *Instance) generatePuzzleList() {
 
 	ret := map[string][]PuzzleMap{}
 	for catName, mb := range ctx.categories {
-		mf, err := mb.Open("map.txt")
-		if err != nil {
-			// File isn't in there
-			continue
-		}
-		defer mf.Close()
-
-		pm := make([]PuzzleMap, 0, 30)
+		filtered_puzzlemap := make([]PuzzleMap, 0, 30)
 		completed := true
-		scanner := bufio.NewScanner(mf)
-		for scanner.Scan() {
-			line := scanner.Text()
 
-			var pointval int
-			var dir string
+		for _, pm := range mb.puzzlemap {
+			filtered_puzzlemap = append(filtered_puzzlemap, pm)
 
-			n, err := fmt.Sscanf(line, "%d %s", &pointval, &dir)
-			if err != nil {
-				log.Printf("Parsing map for %s: %v", catName, err)
-				continue
-			} else if n != 2 {
-				log.Printf("Parsing map for %s: short read", catName)
-				continue
-			}
-
-			pm = append(pm, PuzzleMap{pointval, dir})
-
-			if pointval > maxByCategory[catName] {
+			if pm.Points > maxByCategory[catName] {
 				completed = false
+				maxByCategory[catName] = pm.Points
 				break
 			}
 		}
+
 		if completed {
-			pm = append(pm, PuzzleMap{0, ""})
+			filtered_puzzlemap = append(filtered_puzzlemap, PuzzleMap{0, ""})
 		}
 
-		ret[catName] = pm
+		ret[catName] = filtered_puzzlemap
 	}
 
+	// Cache the unlocked points for use in other functions
+	ctx.MaxPointsUnlocked = maxByCategory
+
 	jpl, err := json.Marshal(ret)
 	if err != nil {
 		log.Printf("Marshalling puzzles.js: %v", err)
@@ -88,13 +68,13 @@ func (ctx *Instance) generatePuzzleList() {
 	ctx.jPuzzleList = jpl
 }
 
-func (ctx *Instance) generatePointsLog() {
+func (ctx *Instance) generatePointsLog(teamId string) []byte {
 	var ret struct {
 		Teams  map[string]string `json:"teams"`
 		Points []*Award          `json:"points"`
 	}
 	ret.Teams = map[string]string{}
-	ret.Points = ctx.PointsLog()
+	ret.Points = ctx.PointsLog(teamId)
 
 	teamNumbersById := map[string]int{}
 	for nr, a := range ret.Points {
@@ -114,9 +94,13 @@ func (ctx *Instance) generatePointsLog() {
 	jpl, err := json.Marshal(ret)
 	if err != nil {
 		log.Printf("Marshalling points.js: %v", err)
-		return
+		return nil
 	}
-	ctx.jPointsLog = jpl
+
+	if len(teamId) == 0 {
+		ctx.jPointsLog = jpl
+	}
+	return jpl
 }
 
 // maintenance runs
@@ -124,6 +108,9 @@ func (ctx *Instance) tidy() {
 	// Do they want to reset everything?
 	ctx.MaybeInitialize()
 
+	// Check set config
+	ctx.UpdateConfig()
+
 	// Refresh all current categories
 	for categoryName, mb := range ctx.categories {
 		if err := mb.Refresh(); err != nil {
@@ -217,17 +204,26 @@ func (ctx *Instance) readTeams() {
 // collectPoints gathers up files in points.new/ and appends their contents to points.log,
 // removing each points.new/ file as it goes.
 func (ctx *Instance) collectPoints() {
-	logf, err := os.OpenFile(ctx.StatePath("points.log"), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
+  points := ctx.PointsLog("")
+
+  pointsFilename := ctx.StatePath("points.log")
+  pointsNewFilename := ctx.StatePath("points.log.new")
+  
+  // Yo, this is delicate.
+  // If we have to return early, we must remove this file.
+  // If the file's written and we move it successfully,
+  // we need to remove all the little points files that built it.
+	newPoints, err := os.OpenFile(pointsNewFilename, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0644)
 	if err != nil {
 		log.Printf("Can't append to points log: %s", err)
 		return
 	}
-	defer logf.Close()
 
 	files, err := ioutil.ReadDir(ctx.StatePath("points.new"))
 	if err != nil {
 		log.Printf("Error reading packages: %s", err)
 	}
+	removearino := make([]string, 0, len(files))
 	for _, f := range files {
 		filename := ctx.StatePath("points.new", f.Name())
 		s, err := ioutil.ReadFile(filename)
@@ -242,7 +238,7 @@ func (ctx *Instance) collectPoints() {
 		}
 
 		duplicate := false
-		for _, e := range ctx.PointsLog() {
+		for _, e := range points {
 			if award.Same(e) {
 				duplicate = true
 				break
@@ -252,13 +248,30 @@ func (ctx *Instance) collectPoints() {
 		if duplicate {
 			log.Printf("Skipping duplicate points: %s", award.String())
 		} else {
-			fmt.Fprintf(logf, "%s\n", award.String())
+		  points = append(points, award)
 		}
+		removearino = append(removearino, filename)
+	}
 
-		logf.Sync()
-		if err := os.Remove(filename); err != nil {
-			log.Printf("Unable to remove %s: %s", filename, err)
-		}
+	sort.Stable(points)
+	for _, point := range points {
+		fmt.Fprintln(newPoints, point.String())
+	}
+	
+	newPoints.Close()
+	
+	if err := os.Rename(pointsNewFilename, pointsFilename); err != nil {
+	  log.Printf("Unable to move %s to %s: %s", pointsFilename, pointsNewFilename, err)
+	  if err := os.Remove(pointsNewFilename); err != nil {
+	    log.Printf("Also couldn't remove %s: %s", pointsNewFilename, err)
+	  }
+	  return
+	}
+
+	for _, filename := range removearino {
+  	if err := os.Remove(filename); err != nil {
+  		log.Printf("Unable to remove %s: %s", filename, err)
+  	}
 	}
 }
 
@@ -286,6 +299,20 @@ func (ctx *Instance) isEnabled() bool {
 	return true
 }
 
+func (ctx *Instance) UpdateConfig() {
+	// Handle export manifest
+	if _, err := os.Stat(ctx.StatePath("export_manifest")); err == nil {
+		if !ctx.Runtime.export_manifest {
+			log.Print("Enabling manifest export")
+			ctx.Runtime.export_manifest = true
+		}
+	} else if ctx.Runtime.export_manifest {
+		log.Print("Disabling manifest export")
+		ctx.Runtime.export_manifest = false
+	}
+
+}
+
 // maintenance is the goroutine that runs a periodic maintenance task
 func (ctx *Instance) Maintenance(maintenanceInterval time.Duration) {
 	for {
@@ -294,7 +321,7 @@ func (ctx *Instance) Maintenance(maintenanceInterval time.Duration) {
 			ctx.readTeams()
 			ctx.collectPoints()
 			ctx.generatePuzzleList()
-			ctx.generatePointsLog()
+			ctx.generatePointsLog("")
 		}
 		select {
 		case <-ctx.update:

cmd/mothdv3/mothd.go

@@ -58,6 +58,12 @@ func main() {
 		20*time.Second,
 		"Time between maintenance tasks",
 	)
+	flag.BoolVar(
+		&ctx.UseXForwarded,
+		"x-forwarded-for",
+		false,
+		"Emit IPs from the X-Forwarded-For header in logs, when available, instead of the source IP. Use this when running behind a load-balancer or proxy",
+	)
 	listen := flag.String(
 		"listen",
 		":8080",

devel.sh

@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# Script to clone and start a development server
-
-set -e
-
-if [ -f tools/devel-server.py ]; then
-	cat <<EOM
-This script is intended to be used to bootstrap a moth development server. It
-looks like you're running the script from a moth repository working directory.
-
-    $ mkdir /tmp/moth
-    $ cd /tmp/moth
-    $ curl https://raw.githubusercontent.com/dirtbags/moth/master/devel.sh | sh
-EOM
-	exit 1
-fi
-
-[ -d puzzles  ] || mkdir -p puzzles
-[ -d moth/bin ] || git clone https://github.com/dirtbags/moth.git
-
-cd moth
-puzzles="$(readlink -e ../puzzles)"
-ln -sf "${puzzles}" puzzles
-
-printf "\n[+] Place puzzles at ${puzzles} ...\n"
-python3 tools/devel-server.py

devel/devel-server.py

@@ -1,6 +1,5 @@
 #!/usr/bin/python3
 
-import asyncio
 import cgitb
 import html
 import cgi
@@ -43,6 +42,13 @@ class MothRequestHandler(http.server.SimpleHTTPRequestHandler):
         except TypeError:
             super().__init__(request, client_address, server)
 
+    # Why isn't this the default?!
+    def guess_type(self, path):
+        mtype, encoding = mimetypes.guess_type(path)
+        if encoding:
+            return "%s; encoding=%s" % (mtype, encoding)
+        else:
+            return mtype
 
     # Backport from Python 3.7
     def translate_path(self, path):
@@ -91,7 +97,7 @@ class MothRequestHandler(http.server.SimpleHTTPRequestHandler):
             "status": "success",
             "data": {
                "short": "",
-               "description": "Provided answer was not in list of answers"
+               "description": "%r was not in list of answers" % self.req.get("answer")
             },
         }
 
@@ -142,6 +148,7 @@ class MothRequestHandler(http.server.SimpleHTTPRequestHandler):
         obj["hint"] = puzzle.hint
         obj["summary"] = puzzle.summary
         obj["logs"] = puzzle.logs
+        obj["format"] = puzzle._source_format
 
         self.send_json(obj)
     endpoints.append(('/{seed}/content/{cat}/{points}/puzzle.json', handle_puzzle))
@@ -252,7 +259,7 @@ if __name__ == '__main__':
         '--theme', default='theme',
         help="Directory containing theme files")
     parser.add_argument(
-        '--bind', default="127.0.0.1:8080",
+        '--bind', default=":8080",
         help="Bind to ip:port"
     )
     parser.add_argument(
@@ -267,7 +274,7 @@ if __name__ == '__main__':
     )
     args = parser.parse_args()
     parts = args.bind.split(":")
-    addr = parts[0] or "0.0.0.0"
+    addr = parts[0]
     port = int(parts[1])
     if args.verbose >= 2:
         log_level = logging.DEBUG
@@ -278,6 +285,8 @@ if __name__ == '__main__':
     
     logging.basicConfig(level=log_level)
     
+    mimetypes.add_type("application/javascript", ".mjs")
+
     server = MothServer((addr, port), MothRequestHandler)
     server.args["base_url"] = args.base
     server.args["puzzles_dir"] = pathlib.Path(args.puzzles)

devel/moth.py

@@ -22,14 +22,12 @@ messageChars = b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
 
 LOGGER = logging.getLogger(__name__)
 
-def djb2hash(str):
-    h = 5381
-    for c in str.encode("utf-8"):
-        h = ((h * 33) + c) & 0xffffffff
-    return h
+def sha256hash(str):
+    return hashlib.sha256(str.encode("utf-8")).hexdigest()
 
 @contextlib.contextmanager
 def pushd(newdir):
+    newdir = str(newdir)
     curdir = os.getcwd()
     LOGGER.debug("Attempting to chdir from %s to %s" % (curdir, newdir))
     os.chdir(newdir)
@@ -123,10 +121,13 @@ class Puzzle:
 
         super().__init__()
 
+        self._source_format = "py"
+
         self.points = points
         self.summary = None
         self.authors = []
         self.answers = []
+        self.xAnchors = {"begin", "end"}
         self.scripts = []
         self.pattern = None
         self.hint = None
@@ -153,8 +154,10 @@ class Puzzle:
         line = ""
         if stream.read(3) == "---":
             header = "yaml"
+            self._source_format = "yaml"
         else:
             header = "moth"
+            self._source_format = "moth"
 
         stream.seek(0)
 
@@ -210,6 +213,16 @@ class Puzzle:
             if not isinstance(val, str):
                 raise ValueError("Answers must be strings, got %s, instead" % (type(val),))
             self.answers.append(val)
+        elif key == 'x-answer-pattern':
+            a = val.strip("*")
+            assert "*" not in a, "Patterns may only have * at the beginning and end"
+            assert "?" not in a, "Patterns do not currently support ? characters"
+            assert "[" not in a, "Patterns do not currently support character ranges"
+            self.answers.append(a)
+            if val.startswith("*"):
+                self.xAnchors.discard("begin")
+            if val.endswith("*"):
+                self.xAnchors.discard("end")
         elif key == "answers":
             for answer in val:
                 if not isinstance(answer, str):
@@ -233,15 +246,31 @@ class Puzzle:
             except IndexError:
                 pass
             self.files[name] = PuzzleFile(stream, name, not hidden)
-        elif key == 'files':
-            for file in val:
-                path = file["path"]
-                stream = open(path, "rb")
-                name = file.get("name") or path
-                self.files[name] = PuzzleFile(stream, name, not file.get("hidden"))
+
+        elif key == 'files' and isinstance(val, dict):
+            for filename, options in val.items():
+                if not options:
+                    options = {}
+                source = options.get("source", filename)
+                hidden = options.get("hidden", False)
+
+                stream = open(source, "rb")
+                self.files[filename] = PuzzleFile(stream, filename, not hidden)
+
+        elif key == 'files' and isinstance(val, list):
+            for filename in val:
+                stream = open(filename, "rb")
+                self.files[filename] = PuzzleFile(stream, filename)
+
         elif key == 'script':
             stream = open(val, 'rb')
             self.add_script_stream(stream, val)
+
+        elif key == "scripts" and isinstance(val, list):
+            for script in val:
+                stream = open(script, "rb")
+                self.add_script_stream(stream, script)
+
         elif key == "objective":
             self.objective = val
         elif key == "success":
@@ -307,6 +336,11 @@ class Puzzle:
             name = self.random_hash()
         self.files[name] = PuzzleFile(stream, name, visible)
 
+    def create_stream(self, name=None, visible=True):
+        stream = io.BytesIO()
+        self.add_stream(stream, name, visible)
+        return stream
+
     def add_file(self, filename, visible=True):
         fd = open(filename, 'rb')
         name = os.path.basename(filename)
@@ -396,7 +430,12 @@ class Puzzle:
         self.body.write('</pre>')
 
     def get_authors(self):
-        return self.authors or [self.author]
+        if len(self.authors) > 0:
+            return self.authors
+        elif hasattr(self, "author"):
+            return [self.author]
+        else:
+            return []
 
     def get_body(self):
         return self.body.getvalue()
@@ -408,7 +447,7 @@ class Puzzle:
     def package(self, answers=False):
         """Return a dict packaging of the puzzle."""
 
-        files = [fn for fn,f in self.files.items() if f.visible]
+        files = sorted([fn for fn,f in self.files.items() if f.visible])
         hidden = [fn for fn,f in self.files.items() if not f.visible]
         return {
             'authors': self.get_authors(),
@@ -422,12 +461,13 @@ class Puzzle:
             'success': self.success,
             'solution': self.solution,
             'ksas': self.ksas,
+            'xAnchors': list(self.xAnchors),
         }
 
     def hashes(self):
         "Return a list of answer hashes"
 
-        return [djb2hash(a) for a in self.answers]
+        return [sha256hash(a) for a in self.answers]
 
 
 class Category:
@@ -460,7 +500,6 @@ class Category:
             with pushd(self.path):
                 self.catmod.make(points, puzzle)
         else:
-            with pushd(self.path):
                 puzzle.read_directory(path)
         return puzzle
 

devel/mothballer.py

@@ -2,12 +2,14 @@
 
 import argparse
 import binascii
+import datetime
 import hashlib
 import io
 import json
 import logging
 import moth
 import os
+import platform
 import shutil
 import tempfile
 import zipfile
@@ -61,6 +63,24 @@ def build_category(categorydir, outdir):
     zipfileraw.close()
     shutil.move(zipfileraw.name, zipfilename)
 
+def write_metadata(ziphandle, category):
+    metadata = {"platform": {}, "moth": {}, "category": {}}
+    
+    try:
+        with open("../VERSION", "r") as infile:
+            version = infile.read().strip()
+            metadata["moth"]["version"] = version
+    except IOError:
+        pass
+
+    metadata["category"]["build_time"] = datetime.datetime.now().strftime("%c")
+    metadata["category"]["type"] = "catmod" if category.catmod is not None else "traditional"
+    metadata["platform"]["arch"] = platform.machine()
+    metadata["platform"]["os"] = platform.system()
+    metadata["platform"]["version"] = platform.platform()
+    metadata["platform"]["python_version"] = platform.python_version()
+
+    ziphandle.writestr("meta.json", json.dumps(metadata))
 
 # Returns a file-like object containing the contents of the new zip file
 def package(categoryname, categorydir, seed):
@@ -90,6 +110,7 @@ def package(categoryname, categorydir, seed):
     zf.writestr("puzzles.txt", "\n".join(str(p) for p in puzzles) + "\n")
     write_kv_pairs(zf, 'answers.txt', answers)
     write_kv_pairs(zf, 'summaries.txt', summary)
+    write_metadata(zf, cat)
 
     # clean up
     zf.close()

doc/devel-server.md

@@ -28,6 +28,7 @@ If you can't use docker,
 try this:
 
 	apt install python3
+	pip3 install scapy pillow PyYAML
 	git clone https://github.com/dirtbags/moth/
 	cd moth
 	python3 devel/devel-server.py --puzzles example-puzzles

example-puzzles/example/2/puzzle.moth

@@ -3,6 +3,7 @@ Summary: Static puzzle resource files
 File: salad.jpg s.jpg
 File: salad2.jpg s2.jpg hidden
 Answer: salad
+X-Answer-Pattern: *pong
 
 You can include additional resources in a static puzzle,
 by dropping them in the directory and listing them in a `File:` header field.

example-puzzles/example/200/puzzle.py

@@ -0,0 +1,19 @@
+import io
+import categorylib  # Category-level libraries can be imported here
+
+def make(puzzle):
+    import puzzlelib  # puzzle-level libraries can only be imported inside of the make function
+    puzzle.authors = ['donaldson']
+    puzzle.summary = 'more crazy stuff you can do with puzzle generation using Python libraries'
+
+    puzzle.body.write("## Crazy Things You Can Do With Puzzle Generation (part II)\n")
+    puzzle.body.write("\n")
+    puzzle.body.write("The source to this puzzle has some more advanced examples of stuff you can do in Python.\n")
+    puzzle.body.write("\n")
+    puzzle.body.write("1 == %s\n\n" % puzzlelib.getone(),)
+    puzzle.body.write("2 == %s\n\n" % categorylib.gettwo(),)
+
+    puzzle.answers.append('tea')
+    answer = puzzle.make_answer()        # Generates a random answer, appending it to puzzle.answers too
+    puzzle.log("Answers: {}".format(puzzle.answers))
+

example-puzzles/example/200/puzzlelib.py

@@ -0,0 +1,7 @@
+"""This is an example of a puzzle-level library.
+
+This library can be imported by sibling puzzles using `import puzzlelib`
+"""
+
+def getone():
+    return 1

example-puzzles/example/4/puzzle.moth

@@ -1,6 +1,8 @@
 Summary: Answer patterns
 Answer: command.com
 Answer: COMMAND.COM
+X-Answer-Pattern: PINBALL.*
+X-Answer-Pattern: pinball.*
 Author: neale
 Pattern: [0-9A-Za-z]{1,8}\.[A-Za-z]{1,3}
 

example-puzzles/example/5/draggable.js

@@ -0,0 +1,75 @@
+// jshint asi:true
+
+var dragSrcEl_
+
+function draggableHandleDragStart(e) {
+  e.target.dataset.moveId = e.timeStamp.toString()
+  e.dataTransfer.effectAllowed = 'move'
+  e.dataTransfer.setData('text/plain', e.target.dataset.moveId)
+
+  // this/e.target is the source node.
+  e.target.classList.add('moving')
+}
+
+function draggableHandleDragOver(e) {
+  if (e.target.attributes.draggable) {
+    e.preventDefault() // Allows us to drop.
+  }
+
+  e.dataTransfer.dropEffect = 'move'
+
+  return false
+}
+
+function draggableHandleDragEnter(e) {
+  e.target.classList.add('over')
+}
+
+function draggableHandleDragLeave(e) {
+  // this/e.target is previous target element.
+  e.target.classList.remove('over')
+}
+
+function draggableHandleDrop(e) {
+  // this/e.target is current target element.
+  let tgt = e.target
+  let src = document.querySelector("[data-move-id=\"" + e.dataTransfer.getData("text/plain") + "\"]")
+
+  // Don't do anything if we're dropping on the same column we're dragging.
+  if (src == tgt) {
+    return false
+  }
+  
+  let tgtPrev = tgt.previousSibling
+  src.replaceWith(tgt)
+  tgtPrev.after(src)
+  
+  tgt.dispatchEvent(new InputEvent("input", {bubbles: true}))
+}
+
+function draggableHandleDragEnd(e) {
+  // this/e.target is the source node.
+  for (e of document.querySelectorAll("[draggable].over")) {
+    e.classList.remove("over")
+  }
+  for (e of document.querySelectorAll("[draggable].moving")) {
+    e.classList.remove("moving")
+  }
+}
+
+function sortableInit(event) {
+  for (let e of document.querySelectorAll("[draggable]")) {
+    e.addEventListener('dragstart', draggableHandleDragStart, false)
+    e.addEventListener('dragenter', draggableHandleDragEnter, false)
+    e.addEventListener('dragover', draggableHandleDragOver, false)
+    e.addEventListener('dragleave', draggableHandleDragLeave, false)
+    e.addEventListener('drop', draggableHandleDrop, false)
+    e.addEventListener('dragend', draggableHandleDragEnd, false)
+  }
+}
+
+if (document.readyState === "loading") {
+  document.addEventListener("DOMContentLoaded", sortableInit)
+} else {
+  sortableInit()
+}

example-puzzles/example/5/helpers.js

@@ -78,6 +78,7 @@ function helperActivate(e) {
   }
 }
 
+
 function helperInit(event) {
   for (let e of document.querySelectorAll(".answer")) {
     helperActivate(e)

example-puzzles/example/5/puzzle.moth

@@ -1,6 +1,7 @@
 Summary: Using JavaScript Input Helpers
 Author: neale
 Script: helpers.js
+Script: draggable.js
 Answer: helper
 
 MOTH only takes static answers:
@@ -42,6 +43,13 @@ Free input, sorted, concatenated values
   <li><button class="expand" title="Add another input">➕</button><l/i>
 </ul>
 
+User-draggable values
+<ul class="answer">
+  <li draggable="true"><input value="First" readonly></li>
+  <li draggable="true"><input value="Third" readonly></li>
+  <li draggable="true"><input value="Second" readonly></li>
+</ul>
+
 Select from an ordered list of options
 <ul class="answer">
   <li><input type="checkbox" value="horn">Horns</li>

example-puzzles/example/categorylib.py

@@ -0,0 +1,7 @@
+"""This is an example of a category-level library.
+
+This library can be imported by child puzzles using `import categorylib`
+"""
+
+def gettwo():
+    return 2