Divide and map. Now. -- square locking policy

I am writing (again) about the Divide and map. Now. The idea of the project is to divide up a big area to organize mapping better. In this note, I will share how I was thinking about the mapping workflow from the beginning, how I finally improved it, and how I hesitated but broke the workflow in the end. And I am sure I did right.

The mapping workflow

The workflow copies a mapathon. Many guys are mapping one area. Few of them are reviewing the work of colleagues. They are locking squares of the area to avoid rewriting the data under their hands.

Since the beginning, it’s possible to lock random or recent square. Mapping guys lock random squares. The reviewing guys lock recent ones to provide feedback to the mappers as soon as possible.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Implementation details
----------------------

Feel free to skip code notes if you are not interested in the code.

There is some background I need to share before showing the SQL query: 

- The server is RESTful.
- Changes to squares work the same as git commits.

(It means that the first commit of any square is `to map`. Then mapper locks a random square adding the `locked` commit to the database. When finished, the mapper adds the `to review` commit. Then the `locked` commit is added again by a reviewer who finally adds the `done` commit.)

The important thing is that no one can delete a commit. Just add. Also, there is no square state. Square consists only of area identifier, square identifier, and border. If you need to know the square's "state," you need to look at the square's last commit.

Finally, the query to map random square is:

    WITH last AS (
	SELECT DISTINCT ON (sid) cid, sid, aid, type, author
	FROM current_commits
	WHERE aid=$1
	ORDER BY sid, cid DESC
    ),
    tomap AS (
	SELECT *
	FROM last
	WHERE type='to map'
	ORDER BY RANDOM()
	LIMIT 1
    )
    INSERT INTO current_commits (sid, aid, author, type, message)
    SELECT sid, $1, $2, 'locked', 'I am mapping'
    FROM tomap
    RETURNING sid

Lock the nearest square

The first enhancement I wanted to implement is locking the square that is the nearest to the one I just mapped. The reason is obvious – I guess that some mappers prefer continuous work. Jumping over the map randomly when the mapper found a nice area could be annoying.

Fortunately, with the PostGIS database, it’s not a big deal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    WITH last AS (
        SELECT DISTINCT ON (cc.sid)
            cid, cc.sid, cc.aid, type, author, border
        FROM current_commits AS cc
        INNER JOIN current_squares AS cs
        ON (cc.aid=cs.aid AND cc.sid=cs.sid)
        WHERE cc.aid=$1
        ORDER BY cc.sid, cid DESC
    ),
    mine AS (
        SELECT border
        FROM current_commits AS cc
        INNER JOIN current_squares AS cs
        ON (cc.aid=cs.aid AND cc.sid=cs.sid)
        WHERE cc.aid=$1
        AND author=$2
        ORDER BY cid DESC
        LIMIT 1
    ),
    tomap AS (
        SELECT sid, cid, type
        FROM last, mine
        WHERE type='to map'
        ORDER BY ST_Distance(mine.border, last.border)
        LIMIT 1
    )
    INSERT INTO current_commits (sid, aid, author, type, message)
    SELECT sid, $1, $2, 'locked', 'I am mapping'
    FROM tomap
    RETURNING sid

Newbie mappers

No doubt, the option to review squares of newbie mappers is favorable.

I am for a pro-active, open approach. (Therefore, any mapper may translate the description of an area.) Therefore, I waited a long time before implementing the option to review a newbie’s square. I can’t say I like the idea the newbie is someone with a small number of squares mapped. Neither I think that advanced mapper marked 43 986 squares to review. (In recent times, this could be discrimination.) And I wasn’t able to come up with something better.

The question is how to find a newbie mapper when the statistics are forbidden/unusable? The answer is pro-active approach. In the last version of the client, I implemented the I am new -> Make myself newbie for next two weeks option. The locking policy for the newbie mapper’s square is then (again) only one SQL query.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
    WITH last AS (
        SELECT DISTINCT ON (sid) cid, sid, type, author
        FROM current_commits
        WHERE aid=$1
        ORDER BY sid, cid DESC
    ),
    toreview AS (
        SELECT sid
        FROM last
        INNER JOIN users
        ON (last.author=users.display_name)
        WHERE type='to review'
        AND (info::json->>'newbie')::timestamp > now()
        ORDER BY RANDOM()
        LIMIT 1
    )
    INSERT INTO current_commits (sid, aid, author, type, message)
    SELECT sid, $1, $2, 'locked', 'Working on review'
    FROM toreview
    RETURNING sid

Breaking the workflow

Good enough for now. The workflow works when everything is all right. I mean – mappers map and send squares for review. Reviewers send them back if more mapping is needed. And mark the squares done otherwise.

But problems happen. And the workflow doesn’t consider making already done square ready for mapping again. Just in case you make a mistake.

The solution is the manual locking of a square. I am wondering why I thought it would introduce troubles. Of course, you can lock multiple squares, but why be afraid of it?

Moreover, when you lock multiple squares, you can merge them. (It means you add done commits for all your locked squares and create a new square to map from the borders of the done ones.) It’s not a hypothetical use-case. It’s implemented. Check the client.

EDIT: After the refactoring, the manual square locking is possible in the mappy client.

Conclusion

I wanted to implement more than random and recent square locking policy since the beginning. The nearest one was clear. And I finally found the approach I like for the newbie’s locking policy.

I acknowledge the importance of manual square locking. However, I don’t think it should be the preferred workflow. Manual locking introduces more control over the square average mapper needs. That’s why I hesitated, I think. Curiosity note: Implementation of merge locked squares functionality took me about 10 hours since I decided to implement till the merge commit.

There are two things I have in mind yet before the end of the year. First and most important, the damn JOSM plugin is reported broken. I need to fix it. The second is that I would like to make creating areas in the current manager a little bit more user friendly. The JSON format is not as comfortable as the HTML form.