Hi my idea for mouse recordings is as follows:
1) Make an app which shows gui in the full screen divided into grid. There should be more cells in the areas where buttons are (fold, raise etc) because bot mostly moves mouse between buttons.
2) During mouse recording, app should position a mouse at a grid cell, and highlight another cell where human should move the mouse. When human reaches that cell, I would store the mouse path (mouse locations + timestamp for each location) in the file (json file). For each recorded mouse path, affine transformations could be apply (such as scaling each point, translating each point, rotating each point) and then storing it in file). That way with affine transformation from one path we could generate path almost between any 2 points.
3) When bot is playing, and wants to move mouse between points P and Q, it should look in the file to which grid cell P belongs and to which Q point belongs. And then find a path between those two cells and load it and perform movement. Cells where buttons are I would make max 0.5x0.5 cm size to be very precise in that area and away from buttons maybe 5x5 cm. If bot wants to move mouse betwen P1 = (300,400) and Q1 = (500,600) but we have the closest points in the file P2 = (302, 401) and Q1 =(502, 603) then first I would move a mouse from P1 to P2 using bezier curve (simulated human - like mouse movement) which would be only by few pixels, and then we have exact mouse starting and ending points as in recorded movement and can perform original human movement.
4) Probably it would be enough to store e.g. 1000 different human mouse movements and then if you need to move mouse between points P and Q, you could transform any human path e.g. points Z to X using affine transformations to points P and Q and perform movement. Affine transformations you can find
https://en.wikipedia.org/wiki/Affine_transformationso the main point is that the shape of the human movements still looks the same. I am not still sure about 4) I would need to test it, but I think that it should work.