Blog / Use cases
Programming a robot on a timeline
On a factory floor, the person who programs a collaborative robot is rarely a programmer. The interfaces that work treat a program as a row of steps: drag Pick, Place, Close lid onto a line, set each one in a popup, press play. We built one as a Grafloria demo, the robot cell programmer, to find out what a timeline editor asks of a diagram engine. This is how it schedules, how it checks a program before the arm moves, how the arm is posed, and the bug our own tests could not see.
What it does
The page has two modes. Simple is one line for one arm: pick a box base, place
it on the work pallet, pick an insert and push it in, fetch a lid and close it, stamp the
box, carry it to the outfeed pallet. A purple Repeat × 6 frame runs those
nine steps once per outfeed slot. Advanced is four lanes on a time ruler: a
conveyor, two arms and a vision camera. A card is as wide as its duration, and a dashed
line between two cards in different lanes means "this one starts only once that one has
finished". Press Run and a six-axis cobot in the middle of the page does the job in 3D
while the cards light up in turn.
Everything a worker edits is the diagram: cards are nodes, loop frames are nodes behind them, the waits are links drawn with the canvas's own connect gesture. Every edit, move and delete is one step on the engine's undo stack.
A timeline is a schedule
In Advanced mode a card's place on the ruler is not where someone dropped it; it is
computed. One rule decides it: a step starts when the step before it in its lane has
finished and every step it waits for has finished. A loop runs its steps
times times in a row, so it ends after times × body. Written
out, it is a short recursive function:
// When does an item end? After its lane predecessor and everything it waits for.
const endOf = (id) => {
if (at.has(id)) return at.get(id).end;
if (visiting.has(id)) { stuck.add(id); return Infinity; } // a wait in a circle
visiting.add(id);
let start = prevOf(id) ? endOf(prevOf(id)) : 0;
for (const w of waitsOf(id)) start = Math.max(start, endOf(w.from));
visiting.delete(id);
if (!Number.isFinite(start)) { stuck.add(id); return Infinity; }
const end = start + secondsOf(item(id)); // a loop: times × its body
at.set(id, { start, end });
return end;
};
The interesting line is the circle. If the camera waits for the stamp and the stamp waits for the camera, neither can ever start. A naive scheduler loops forever or picks an arbitrary order; this one marks both as stuck, parks them at the end of their lanes and says "this waits for something that waits for it". The editor goes further and refuses to draw such a wait in the first place, using the engine's connection validator, which also refuses a wait inside one lane (those steps already run in order):
registerConnectionValidator(({ sourceNode, targetNode }) => {
if (laneOf(sourceNode) === laneOf(targetNode)) return 'Steps in one lane already run in order';
const trial = { ...program, syncs: [...syncs, { from: sourceNode.id, to: targetNode.id }] };
if (schedule(trial).stuck.length) return 'They would wait for each other forever';
return true;
});
Because the schedule is derived, the layout follows it: delete the camera's wait for the stamp and its card slides to the start of the ruler, with a hatched bar under the lanes that are now idle.
Checking the program before the arm moves
A program can be valid as a diagram and wrong as a job. Place with an empty gripper, Insert before there is a box to insert into, a seventh box for a six-slot pallet. The demo finds these by replaying the schedule through a small model of the cell: what each gripper holds, what is on the work pallet, whether a box waits at the conveyor stop, which outfeed slots are full. Each step has a precondition, checked when it starts, and an effect, applied when it ends. Applying effects at the end matters as soon as two lanes overlap in time: the camera may not inspect a stamp that is still being printed.
case 'insert':
if (!held) return { level: 'error', msg: 'The gripper is empty. Pick an insert first.' };
if (held.kind !== 'insert') return { level: 'error', msg: `The gripper holds ${a(thing(held))}, not an insert.` };
if (!cell.work || cell.work.lid) return { level: 'error', msg: 'There is no open box on the work pallet to insert into.' };
if (cell.work.inserts >= 4) return { level: 'error', msg: 'The box is full: it takes four inserts.' };
return null;
Our first version reported every problem it met. Move the insert's Pick after the Insert and the board turned red nine times: the Insert had an empty gripper, but on the next pass the arm was still holding the insert, so the first Pick "already held an insert", the Place tried to put an insert on the pallet, and so on around the loop. Every message was true and eight of them were noise. A real cell stops at its first fault, so the replay now does too, and the board shows one red card with the cause. Warnings, such as a box leaving without a stamp, do not stop the replay and do not block Run.
A pose is a function of time
The 3D arm is three.js, but the way it is driven is the part worth copying. There is no
animation state. Given the run's events and a time t, the arm's six joint
angles are computed from scratch: where its last finished step left it, then the running
step's profile. A pick reaches above the station for the first 45%, descends until 60%,
closes its fingers until 72%, and rises. The joints for a target come from a closed-form
inverse kinematics: turn the base toward the target, solve the shoulder and elbow as a
two-link arm in that plane, and tilt the wrist so the tool keeps pointing straight
down.
function solve(base, x, y, z) {
const a1 = Math.atan2(-(z - base.z), x - base.x); // turn to face the target
const r = Math.hypot(x - base.x, z - base.z);
const h = y + TOOL - SHOULDER_HEIGHT; // the wrist sits TOOL above the tips
const d = clamp(Math.hypot(r, h), Math.abs(L1 - L2) + 1e-3, L1 + L2 - 1e-3);
const e = Math.acos((d * d - L1 * L1 - L2 * L2) / (2 * L1 * L2));
const s = Math.atan2(h, r) + Math.atan2(L2 * Math.sin(e), L1 + L2 * Math.cos(e)); // elbow up
return [a1, s, -e, 0, -Math.PI / 2 - (s - e), 0]; // the wrist keeps the tool vertical
}
Making the pose a pure function of t buys three things. Step and
seeking need no special code: set t, draw. A reload mid-run draws the same
arm. And the pose is testable. The page's own checks seek to 60% of the first Pick and
require the finger tips to be within 3 cm of the stack. To make sure that check could
fail, we gave the arm a wrong upper-arm length for one run: the tips missed by 4.7 cm
and the check went red. A check you have never seen fail is a check you have to take on
trust.
What we got wrong
Drops landed in the wrong place, and every test passed. To turn a mouse position
into canvas coordinates, the page first used viewport.x + px / zoom, copied
from our own drag-and-drop demo. That is only right at zoom 1: the camera zooms about the
middle of the view, so viewport.x is not the world's left edge once the zoom
changes. Advanced mode opened at 0.72, where a drop landed about 180 canvas pixels from the
pointer. The checks passed because they converted the other way with the same helper: a
formula that is wrong in both directions agrees with itself. The fix was one line,
api.viewport.clientToWorld(x, y, rect), the engine's exact inverse of what it
draws. The lesson took longer. The checks now take their drop points from where the cards
are on screen, and with the old formula put back for one run, the drag check fails as it
should. The drag-and-drop demo had the same bug, 224 pixels off at zoom 0.6; it is fixed,
with a check that failed first.
Two selections. Cards are the page's to delete: a card leaves the program, its waits go with it, and one undo brings both back. But a card selected through the engine, by a marquee or a script, was deleted by the engine directly, so the canvas and the program disagreed. There is now one selection, kept in step with the engine's, and Delete always goes through the program.
The parts
The cards are HTML inside nodes. Grafloria takes node HTML as a sanitised tree, not a
string: plain tags only, no inline SVG and no data- attributes, so the icons
are CSS masks and a card is found by its node's data-node-id. The timeline
is docked along the bottom of the page and behaves like one: it fills the dock's height,
the wheel scrolls it sideways, and it stops at the program's ends. The settings popups are
Fieldia forms, the 3D cell is
three.js, and the catalog, scheduler, replay, layout, cards and kinematics are about thirteen hundred lines of
framework-free JavaScript next to the page.
Open the robot cell
programmer, break the program on purpose (drag the insert's Pick after the Insert, or
set the loop to seven), then switch to Advanced and delete one of the dashed waits. It
runs on @grafloria/element and @grafloria/renderer 0.7, MIT.