GPC: Mixed Real/Virtual Articulated Robot Simulator (MSim)

ogv|mp4|webm

Interactively adding a virtual revolute joint to facilitate tilting the field of view of a side-facing camera on NASA/JPL's ATHLETE. ATHLETE model courtesy of RSVP Team NASA/JPL/Caltech. See the MSim-ATHLETE page for more examples with ATHLETE.

The Mixed Real/Virtual Articulated Robot Simulator MSim is is a general-purpose interactive graphical simulator for 3D articulated robots, i.e., systems composed of rigid-body links and joints that connect one body to another. MSim was originally developed by Prof. Vona for his dissertation Virtual Articulation and Kinematic Abstraction in Robotics (Ph.D., MIT EECS, 2009).

MSim simulates kinematics, including closed and open chains and joint limits, with a catalog of 11 joint types including all the lower pair joints except helical; gravitational potential energy stored in links with mass; and elastic potential energy stored in joints with stiffness. By allowing kinematic structure to be changed arbitrarily and dynamically, MSim supports research on virtual articulation—the idea of virtually modifying the kinematic structure of an articulated robot, for example, by adding extra joints and links.

MSim is released under the GNU GPL. You can also browse the source and read the javadoc online. More info on downloading and known issues below.

Operating High-DoF Robots

Modular Towerogv|mp4|webm

A simulated modular robot tower with over 270 joints and 90 kinematic cycles.

MSim enables exploration of some new ideas in several robotics domains. In one area, virtual articulations can help build a richer semantics for specification and communication of coordinated joint motions in arbitrary-topology high-Degree-of-Freedom (DoF) robots. With collaborators at NASA/JPL, MSim has been used to demonstrate the rapid specification of coordinated motions for ATHLETE, the All-Terrain Hex-Limbed Extra-Terrestrial Explorer, a research platform under development at JPL for potential use in aiding humans in future Lunar missions. The animation at the top of this page, recorded directly in MSim, shows a virtual revolute joint added to facilitate tilting the field of view of one of ATHLETE's side cameras. The virtual joint closes a kinematic chain between the ATHLETE deck and the ground link, constraining the deck to a 1D space of orientations that effectively tilt the camera about the desired axis. The entire model can be manipulated subject to this constraint, and the camera tilt can now be explicitly and quantitatively commanded by directly operating the virtual joint. MSim computes feasible motions interactively in real time. The model we use for ATHLETE is purely kinematic, but MSim is also able to handle quasistatic models that include link mass, gravity, and joint stiffness. See the MSim-ATHLETE page for more details on our work for ATHLETE, including further examples of motions (both in simulation and on actual hardware) that would have been difficult to specify in prior systems.

Another way virtual articulations can help operate high-DoF robots is by using them to define abstractions for localized parts of the robot. These structure abstractions can also form a heirarchy, and thus a complicated and detailed articulated assembly can be considered in terms of a simpler overall abstraction. At left is an example of a simulated modular tower with over 270 joints. Interactive simulation speed is maintained by abstracting each tower block as a simpler virtual articulated structure consisting of one revolute and one prismatic joint (these are shown faintly in the animation). This heirarchy was also a convenient way to specify the intended motion, first at the level of each block, and then at the level of the full tower.

Analyzing Compliant Contact

A second domain MSim has been used to explore is the use of virtual articulations to model contacts, compliances, and geometric uncertainties in the environment of compliant robots. We've used it to analyze aspects of the motion of our vertical truss climbing robot and also a stair-stepping mini-humanoid. Both of these robots are designed with joint compliance and rely on proprioceptive control strategies to tolerate uncertainty as they locomote. The virtual articulation models can help understand, analyze, and predict both the limits and the sensitivity of proprioceptive sensing. In some cases a single virtual articulation can effectively model what is in reality a more complicated sequence of contact states. An example of such sequence abstraction is the truss climbing robot's upper gripper contact, which is modeled with a single prismatic joint.

Truss Climbing Uncertainty Toleranceogv|mp4|webm

A simulation of our vertical truss climbing robot in which the robot is making a transition from a vertical to a horizontal bar. Uncertainty in the position of the horizontal bar is simulated by a virtual prismatic joint (highlighted). The joints of the robot will comply to accommodate this uncertainty, and it will also change the ultimate position of the right gripper on the bar. This simulation accurately predicts both the limits and the sensitivities for detecting the uncertainty (via proprioception in the compliant joints), but uses only a greatly simplified model the contact sequence of the right gripper, abstracting it as a prismatic joint.

Stair Climbing Uncertainty Toleranceogv|mp4|webm

A quasi-static simulation of a mini-humanoid robot probing a step of unknown height. The step height is modeled as a virtual prismatic joint, and the contacts of each foot are also modeled with virtual joints. The robot can detect steps within a range of heights. The minimum detectable step is determined by the first contact of the right heel, which occurs when the associated point-slider joint hits a limit. The maximum detectable step is ultimately limited by the stability of the robot. The simulation results for these limits compare well with actual hardware experiments, even though the model is greatly simplified.

Comparison to Other Simulators

Some key features differentiate MSim from other contemporary articulated robot simulators:

A more thorough review of the relevant research context is given below.

MSim is a work in progress. The core topological and numeric data structures and algorithms are in place and operational, as are basic textual and graphical UI components, but many potentially useful features (e.g. collision detection) remain to be implemented. See the roadmap.

Model Structure and Joint Representation

Topologically, the MSim model is a directed graph with vertices corresponding to links and edges corresponding to joints. The model topology may be specified and mutated either graphically or textually—MSim includes both a 3D display and a Scheme interpreter command interface. A spanning tree is always inherently included as part of the model specification. Joints in the spanning tree are called tree joints; the remaining closure joints close kinematic cycles.

Joint Typesogv|mp4|webm

Demonstration of the motion of the 11 currently available joint types in MSim. Top row, left to right: revolute, prismatic, cylindrical, spherical. Middle row: planar, pin-slider, point-slider. Bottom row: point-plane, cartesian-2, cartesian-3, general.

Graph edge directions correspond to the forward sense of the corresponding joint's rigid body transform that takes coordinates in the frame of the the child link to the frame of the parent link. In the current implementation, the edge directions of tree joints are also constrained to point towards the ground link at the base of the tree. By default, MSim renders curves representing tree joints in ultramarine and closure joints in purple. Arrows indicating edge directions are not displayed by default but can be turned on by pressing 'n'.

MSim ensures that every link other than the ground link has exactly one tree joint parent at all times, so the model graph is always connected and the spanning tree determines the current spatial pose of every link relative to the ground link. MSim maintains the one-parent invariant for all links while still allowing arbitrary user-specified topology by automatically managing a set of distinguished joints, one per link, called root joints. The root joint r of a link l is always present, and attaches that link directly to the ground link. Root joints are always of type general, which means they support full 6-DoF mobility. In the common case where l has some (user-specified) tree parent, MSim automatically converts r to a closure joint; if the user should ever orphan l MSim automatically promotes r back to a tree joint. By default, unlocked root closure joints are not rendered to avoid visual clutter.

Separating the kinematic model into a spanning tree and closure joints helps structure and limit the computation of the numeric solver, as detailed below. And semantically, the operator knows that while tree joints must always respect their structure—for example, a revolute tree joint must always stay within its joint angle limits and never permits any translation or off-axis rotation—closure joints may need to break when a closed kinematic chain is inconsistently overconstrained.

MSim's numeric solver employs the common technique of local linearization. In the domain of kinematics this is generally possible because the space of all three-dimensional rigid-body transforms, SE(3), has manifold topology, implying that the space of small local variations infinitesimally near a configuration is always linear (though singularities can be introduced when joints are assembled into closed chains, MSim's solver is designed to handle them). Quasistatics are also handled by local linearization via the derivatives of gravitational and linear spring potential energy functions.

MSim homogenizes all non-topological interaction, including dragging links in the GUI and operating joints either textually or graphically, as interacting with joint configurations. Dragging a link generates an equivalent goal for the configuration of its root joint. In addition, the operator may freeze the pose of any joint by locking it, and consequently a link may be frozen with respect to the root link by locking its root joint. Locked root joints are rendered as red curves; in the example at the top of the page all the wheels have locked root joints. The lock status of any link or joint can be toggled by picking it in the GUI and pressing 'l'; locked joints are rendered in red.

See the Linkage, Joint, and RX Javadoc for additional details on the implementation of model datastructures in MSim.

Simulation Algorithms

Kinematic simulation is trivial when no closure joints are present (the open-chain case): the system simply moves the tree joints as specified by the operator, computing forward kinematics by composing the rigid-body transforms corresponding to the joint configuration. In the quasistatic case, each joint may have an associated potential energy function due to joint elasticity or, in the case of root joints, mass of the attached link. In the open-chain case the system moves such joints to a nearest configuration with minimum potential energy.

When closure joints are present, MSim incrementally adjusts the configurations of the tree joints to satisfy constraints in a cascade of priority levels:

  1. invariant: The closure joints' structure should be maximally respected. Again, take the example of a revolute joint: it should permit no translation and no off-axis rotation.
  2. limit: Each joint DoF may optionally be limited to stay within a specified range. The solver automatically observes these limits for tree joints, but for closure joints with set limits an explicit constraint is introduced that penalizes the joint for exceeding its limits.
  3. lock: The system should hold fixed the full configuration of any operator-specified locked joints.
  4. potential: The system seeks to move joints with associated potential energy functions, which model joint elasticity and link mass, to their minima.
  5. target: When the operator requests that a particular joint move to a particular configuration (e.g. by dragging it, or by dragging the link attached to a root joint), the system creates a target to model the intended configuration.
  6. posture: A default posture may be specified for each joint. This can help break ties when exiting singular configurations, and bias the system to stay within an intended component of the configuration space.
A key feature of MSim is that these goals are solved in priority order, so that, for example, a lock configuration overrides a conflicting target.

Topological Analysis

Not every tree joint necessarily affects the configuration of a given closure joint c. Intuitively, tree joints on un-related branches need not be considered. Formally, we trace the attachment points of c back to the link that is their least common ancestor (LCA); the set of tree joints we pass on the way is the support of c. We identify the support of all chain closures in this way, and if the supports are all disjoint, we are finished. Of course, it is common for supports to overlap; chain closures with overlapping support are generally coupled and need to be solved simultaneously. Thus as a second stage of topological analysis we trace the transitive overlap of supports, forming solve groups of coupled chain closures and their supports. Each solve group is then assigned its own numeric solver. We call this the coupling decomposition. Effectively, we find the biconnected components of the kinematic graph with the edge directions ignored, but we do so in a way that is compatible with a secondary hierarchical decomposition.

In some cases the entire kinematic graph is a single biconnected component, so the coupling decomposition will not have any effect. We introduce a second stage of topological decomposition based on the model's structure as a nested hierarchy of sublinkages. This hierarchy is optional, and when present it is specified by the operator or as part of the model construction. Each sublinkage is specified to be either driving, driven, or simultaneous within its parent, referring to the order in which constraints are solved. Thus the sublinkage containment graph is a DAG, and a topological sort yields a solve order for its strongly connected components (simultaneous sublinkages correspond to bidirected edges, yielding SCCs that may contain more than one sublinkage). The above coupling decomposition is designed to respect sublinkage boundaries, which gives the operator a structured means to limit the extent of coupled solve groups. This power comes at a price: forcibly decoupling a solve group at a driving or driven sublinkage boundary may mean that the second-solved group cannot reach every configuration of the first-solved group. MSim handles this like any other overconstrained case and finds a nearest configuration that approximates constraint satisfaction in a least-squares sense.

Topological analysis is performed automatically whenever the kinematic topology changes, and is generally imperceptible when interacting with MSim.

Numeric Solver

For each solve group MSim (1) computes a Jacobian matrix relating small changes in the variables (tree joint configurations) to small changes in the constraints (closure joint configurations), (2) uses a prioritized damped least squares solver [SS91] [BB04] to invert this relationship, transforming a small output change that improves constraint satisfaction into a small adjustment of the tree joints that will effect this change, and (3) applies the change and iterates until the constraints are sufficiently satisfied.

The entire problem may change dynamically, both in numeric values and in topological structure. Structural changes trigger a topological re-analysis, and numeric value changes are handled by further solve iterations.

There are a number of challenges to consider in implementing the solver:

  • The Jacobian computation must be general, to handle the various combinations of actuator and effector joint types, numerically robust, accurate, and fast, as it needs to be performed before every iteration. MSim leverages the exponential map joint parametrization to form the Jacobian from analytic derivatives of the exponential and log maps composed with the current relative poses of the tree and closure joint mobility spaces.
  • The Jacobian is not generally invertible. The problem may be structurally over- or under-constrained, i.e. there may be more constraints than variables or vice-versa; in either case the Jacobian will not be a square matrix, so it will be structurally singular. Also there may be linear dependencies among the variables and/or among the constraints which further reduce the Jacobian rank. MSim computes an SVD-based pseudoinverse to handle either case in a least-squares optimal sense.
  • Near-singular configurations are numerically ill-conditioned in the pseudoinverse method, which in practice results in instability and oscillation. MSim applies adaptive numeric damping in these cases, which sacrifices speed of convergence for stability. Specification of the optional low-priority joint postures also helps "break the tie" when leaving singular configurations.
  • The linear model embodied in the Jacobian is an approximation which only holds in some locality of the current configuration. MSim uses an adaptive speedup/slowdown algorithm to limit the step size at each iteration.

  • In general the constraints may conflict with each other: maximally satisfying one may require compromise in others. MSim implements a priority formulation where the constraints are partitioned into priority levels, with the highest-priority levels solved first, and the solutions for lower levels projected onto the nullspace of higher levels.

Like topological analysis, solve iterations are automatically scheduled in MSim and are generally transparent to the user: you simply operate joints and drag links and MSim moves everything else to keep the constraints satisfied.

See the Solver class doc for additional details.

Research Context

Though kinematic modeling and simulation is well studied, relatively few implemented systems have been reported in the literature that have all of the following properties:

  • genericity — Any 3D kinematic topology, including under- and over-constrained, can be simulated, with a reasonably large set of available joint types and joint motion limits.
  • mutability — The kinematic topology can be changed on-line.
  • mixed reification — Joints and links are either considered part of an actual mechanism or purely virtual at the user's discretion.
  • pure kinematicity — Though physics simulation is obviously useful and important, in some cases it is not only sufficient but also faster and more intuitive to specify only kinematic properties, especially for virtual constructions.
  • identified spanning tree — The spanning tree not only helps manage the computational cost of simulation, but also becomes semantically meaningful in inconsistently overconstrained cases.
  • prioritized constraints — When inconsistency is encountered it can be useful for the system to maintain some constraints over others.

A few authors, including Fluckiger [Flu98] and Collins, McCarthy et al. [CMPS02] have previously developed generic interactive 3D kinematic simulators. The former included an identified spanning tree, but did not appear to have significant capability for on-line mutation of the kinematic topology or constraint priorities. The latter system was aimed towards automated kinematic synthesis and did not focus on operator interaction. Beyond interactive systems, many lower-level kinematics modeling libraries have been developed; some modern works include Bruyninckx et al's ORCOS/KDL, Diza-Calderon, Nesnas, et al's CLARAty Mechanism Model [CNNK06], Pryor, Tayolor, Kapoor, and Tesar's OSCAR [PTKT02], and the Mathworks' SimMechanics package. While most of these are able to model arbitrary topology kinematic systems, again they are not aimed towards supporting on-line topological mutation (with the notable exception of [PTKT02]), and as low-level libraries they do not generally incorporate user-interface components.

Pseudoinverse-based iteration on a local linear kinematic approximation has a literature in both robotics and computer animation dating back at least to Whitney's description of the basic technique of resolved rate control [Whi69]. Liegeois extended the approach to two priority levels using nullspace projection [Lie77], and Siciliano and Slotine further extended it to an arbitrary number of priorities [SS91]. Finally, Baerlocher and Boulic gave a more efficient recursive formulation for the multi-priority case [BB04]. MSim's Solver is based on this recursive formulation but also adds an adaptive step size component.

Geometric Constriant Solving

Geometric constraint solving is a highly evolved and active field [HA05], and while much of the focus has been on the static case of solving for a single configuration that satisfies a given set of constraints, systems with dynamic user interaction date back as far as Sutherland's original work [Sut63]. Modern 3D CAD systems often have some capability to construct 3D rigid body assemblies and then permit the user to drag individual components while maintaining assembly constraints. In practice these implementations appear to work reasonably well [Hof01] when the model is not over-constrained (typically CAD systems consider overconstraint to be an error) and only mildly under-constrained, so that the system does not need to infer motion for more than a small number of DoF. For highly under-constrained (kinematically redundant) cases, CAD constraint solvers typically infer additional constraints according to heuristics, and this can lead to surprising motions when the inferred constraints are not intuitive.

While it is common to implement kinematic constraint satisfaction with numeric iteration, it is less common to explicitly use virtual links and joints to define the constraints. Ashrafiuon and Sanka [AS96] and Ivlev and Graser [IG98], have proposed virtual joints as a means of aiding inverse kinematic solution algorithms for redundant mechanisms. In these cases the virtual joints are not explicitly specified by the operator to constrain motion or to define task-relevant DoF, but rather are automatically added in a restricted way only to help control algorithmic complexity of inverse kinematic solution, and may not even be known to the operator.

Exposing geometric constraints as a user-interface tool for constraining motion, especially in the field of articulated figure animation, has been considered by several authors including Phillips, Zhao, and Badler [PZB90] and Welman [Wel93]. In these works typically only a few simple constraints are available, for example, a particular point can be constrained to lie on a particular plane. The constraints are not themselves considered to be kinematic joints, and can only be added between geometric objects situated within links of the original articulated model. In MSim the operator is free to construct arbitrary additions including both virtual links and joints. Unlike pure geometric constraints (parallelism, coincidence, etc), virtual joints have the added benefit of defining a concrete parametrization for the constrained motion.

Physics Simulation

Recently there has been a trend to apply physics simulators—e.g. Webots—to articulated mechanisms. This can of course come much closer to predicting the behavior of actual physical dynamic systems, and in fact we may eventually extend MSim to also include full dynamic physics simulation as an option. Currently MSim is primarily a kinematic simulator, with optional ability to model quasistatic effects of (a) gravity on link mass and (b) linear-spring stiffness on 1-DoF joints.

While there is increasing research interest in under-actuated and compliant robots, in practice it is still the case that many robots are fully and rigidly actuated, especially in postural motion and statically stable locomotion and manipulation tasks: their state at any moment is effectively only a function of their joint configurations at that moment. The need for full physics simulation in these contexts is significantly reduced. And because it can simulate purely kinematic models, MSim also neither requires the various parameters—inertia tensors, stiffness matrices, damping factors, friction constants—to be specified or tuned. This can be important when combined with our interest in rapidly changing kinematic topology; it is desirable for the system to behave reasonably without requiring the operator to enter detailed physical information (or even surface geometry) as virtual links and joints are added (of course, a physics simulator could also be configured to behave reasonably when dealing with zero-mass objects, but this would essentially turn it into a kinematic simulator).

By not requiring the simulator to strictly model physics—and again, in practice many robots will be effectively dominating natural dynamics with energy input through their stiff joints—we have room to implement features like prioritized constraints, the spanning tree/closure joint model, and hierarchical decomposition. For fully-actuated robots the computed motions can still be executed because in this case actuation is intended to override natural dynamics.

Of course, if full physics simulation is performed, then one could also consider adding virtual elements that are not strictly kinematic; for example, virtual springs and dampers. This has been explored by Pratt et al [PCTDP01] as virtual model control; in their systems the virtual elements are manually designed, tuned, and implemented on a case-by-case basis for each task of each mechanism, and then simulated as part of the on-line high bandwidth closed loop control computation. Our work focuses instead on the issues that arise in building a system for rapid interactive on-the-fly construction of virtual articulations in arbitrary kinematic topology mechanisms.

Related Publications

Marsette Arthur Vona, III. Virtual Articulation and Kinematic Abstraction in Robotics, Ph.D. Dissertation, Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, September, 2009.

Marsette Vona. Operating High-DoF Articulated Robots Using Virtual Links and Joints, Gulrez, Tauseef; Hassanien, Aboul Ella (Eds.). Advances in Robotics and Virtual Reality, Springer Intelligent Systems Reference Library, Vol. 26, Part III, pp. 275-305, 2012.

Marsette Vona. Hierarchical Decomposition and Kinematic Abstraction with Virtual Articulations, Advances in Robot Kinematics, pp. 33-43, June, 2010.

Marsette Vona. A Graphical Operations Interface for Modular Surface Systems, IEEE Aerospace Conference, March, 2010.

Marsette Vona, David S. Mittman, Jeffrey S. Norris, Daniela Rus. Using Virtual Articulations to Operate High-DoF Manipulation and Inspection Motions, International Conference on Field and Service Robotics, pp. 355-364, July, 2009.

David S. Mittman, Jeffrey S. Norris, Mark W. Powell, Recaredo J. Torres, Christopher McQuin, Marsette A. Vona. Lessons Learned from All-Terrain Hex-Limbed Extra-Terrestrial Explorer Robot Field Test Operations at Moses Lake Sand Dunes, Washington, American Institute of Aeronautics and Astronautics Space Conference, September, 2008.

Interacting with the Model

You interact with MSim both graphically, using keyboard and mouse input in the GUI, and textually, by evaluating Scheme forms in the MSim command shell. Some interactions are topological, for example adding a new link or changing the type of a joint. Other interactions do not change kinematic topology but only manipulate the numeric values in some parts of the existing model.

Topological Interaction

Topological mutations automatically trigger a re-analysis of the model structure. Use (update-protect-[a]sync), abbreviation mup{a,s}, to schedule multiple mutations as a single atomic update.

These are the basic topological mutations currently available in MSim:

  • add sublinkage(cons-linkage [parent [disposition [name]]]) — a new sublinkage is added, see the Linkage class header doc for details on sublinkages
  • add link(cons-link [linkage] [name]), abbrev cl, keybinding 'shift-l' — a new link is added, initially parented to the ground link through its root joint. When initiated through the GUI, the new link appears under the mouse cursor and manipulation is automatically initiated for its pose.
  • add joint(cons-joint [linkage] [name] [type] [parent-link child-link]), abbrev cj, keybinding 'shift-j' — a new joint is added between the specified links (or dangling). When initiated through the GUI, the new joint appears under the mouse cursor and manipulation is automatically initiated for the mobility space pose.
  • remove a sublinkage, link, or joint(remove sublinkage-or-link-or-joint), abbrev rm, keybinding delete — removes the indicated link or joint. Removing a link dangles the adjacent joints; removing a tree joint automatically converts its child's root link to a tree joint. Operates on the currently picked link or joint when initiated through the GUI.
  • set joint parent/child(set-{parent,child}-link joint link), keybindings '<','>' — set the parent or child link of a joint. When initiated through the GUI the parent/child of the currently picked joint is set to the currently selected link.
  • change joint type(set-joint-type joint type), no keybinding — change the mobility type of a joint. For example, change a revolute joint to a cylindrical joint.
  • make tree joint(make-tree-joint joint), abbrev mtj, keybinding 't' — convert a closure joint to a tree joint. The prior parent of the child link is converted to a closure joint. Operates on the currently picked joint when initiated through the GUI.
  • make closure joint(make-closure-joint joint), abbrev mcj, keybinding 'c' — convert a tree joint to a closure joint. The child link's root joint is converted to a tree joint. Operates on the currently picked joint when initiated through the GUI.
  • make root link(make-root-link link), abbrev mrl, keybinding 'r' — same as make-tree-joint on a link's root joint. Operates on the currently picked link when initiated through the GUI.
  • make ground link(make-ground-link link), abbrev mgl, keybinding 'g' — re-grounds the linkage. Operates on the currently picked link when initiated through the GUI.
  • split link(split-link link), abbrev sl, keybinding '*' — split a link by inserting a new joint and link in the spanning tree between the original link and its original parent joint. Operates on the currently picked link when initiated through the GUI.
  • merge joint(merge-joint joint), abbrev mj, keybinding '/' — removes a joint and its child link, re-attaching other joints that were attached the child to the joint's parent link. Operates on the currently picked joint when initiated through the GUI.
  • lock/unlock joint([un]lock-node joint-or-link), keybinding 'l' — locks or unlocks a joint or the root joint of a link. Operates on the currently picked link or joint when initiated through the GUI.
  • invert joint(invert-joint joint), abbrev ij, keybinding '%' — inverts a joint. For a closure joint, this is a topology swap and a geometric inversion. For a tree joint this triggers a re-grounding: the child link of the joint becomes the new ground link, and all tree joints on the path to the old ground link are also inverted. Operates on the currently picked joint when initiated through the GUI.
  • add/remove joint target or posture([un]set-{target,posture} joint [dof...]), keybindings {alt,shift}-m — adds or removes target/posture goal(s) on one or more DoF. Manipulating a joint or link in the GUI will add target and posture goals as necessary, see below.
  • add/remove/change joint limits([un]set-all-dof-limits joint limits...), no keybindings — adds or removes per-DoF upper and lower limits on a joint. Currently even changing the numeric value of a limit is treated as a topological mutation.

Manipulation

MSim uses the term manipulation specifically for interactions that change numeric values in the model but do not change the model's topological structure. Manipulating a particular joint's configuration is a common case (this includes manipulating a link's pose by manipulating its root joint), and is supported in the GUI with a modal interface that re-purposes the arrow keys and mouse from their default navigation functions. You can toggle back and forth to navigation while manipulating a joint by tapping the spacebar.

(update-protect-[a]sync) can also be used to schedule multiple numeric mutations, or an ordered mix of topological and numeric mutations, as a single atomic update.

These are the basic numeric mutations currently available in MSim:

  • change joint configuration(set-mobility joint dof...), abbrev sm, keybinding 'm' — actuate a joint. Note that actuating a joint that is part of a kinematic cycle will generally have little effect due to the action of its solver; instead manipulate its target or posture. Manipulating a picked link in the GUI enters manipulation mode on its root joint, and manipulating a picked joint in a closed chain automatically defaults to manipulating the joint's target. Hit escape to stop manipulating.
  • change existing target or posture goal(set-{target,posture} joint dof...), keybindings {alt,shift}-m — adds a target/posture goal (which is actually a topological mutation) if none is already present, else change the existing goal's value. Similar behavior as above for manipulating in the GUI, but explicitly manipulates target or posture. In the GUI goals are only added when they are changed (so if you only rotate a link which previously had no goals, a rotation goal is set but the translation remains unconstrained). By default the goal will be un-set when manipulation ends, or end manipulation with ctrl-escape to keep the goal.
  • change joint mobility space pose(transform-mobility joint transform), keybinding ctrl-m — manipulate the pose of the mobility space of a joint. For example, change the placement of the rotation axis of a revolute joint.
  • set/unset the gravity vector([un]set-gravity x y z), no keybinding— set or unset the direction and magnitude of the gravity acceleration vector in ultimate ground frame. By default the gravity vector is unset, which globally disables gravitational potential energy simulation.
  • set/unset link mass([un]set-mass link mass), no keybinding— set or unset the mass of a link. By default link masses are unset, which disables gravitational potential energy simulation for that link.
  • set link center of mass(set-center-of-mass link x y z), no keybinding— set the center-of-mass of a link, in link frame. By default link CoMs are at link frame origin.
  • set/unset joint stiffness([un]set-stiffness joint stiffness), no keybinding— set or unset the stiffness of a 1-DoF (revolute or prismatic) joint. By default joint stiffnesses are unset, which disables elastic potential energy simulation for that link.
  • set joint stiffness rest (set-stiffness-rest joint stiffness-rest), no keybinding— set the stiffness rest pose of a 1-DoF (revolute or prismatic) joint. By default stiffness rest poses are zero, or the nearest DoF limit to zero.

A number of variants on these basic operations are also available, see msim-api.scm, msim-extra.scm, and the Joint and Link Javadoc.

Command Shell

The textual interface for interacting with MSim is a REPL-style interactive JScheme interpreter. The full MSim JScheme API is defined in msim-api.scm, with additional scheme sugar in msim-extra.scm. In general, the JScheme api is a thin cover over the corresponding Java API. For example, the JScheme form (dvi) is equivalent to the Java API MSim.dumpVersionInfo(). See the MSim and VonaUtils javadoc (especially vona.math, vona.j3d, and vona.ui) for further details.

When running MSim as an application, e.g. java -jar MSim-newest.jar, the MSim interpreter runs at the console. It may be more convenient to use a console with specific support for scheme interaction, for example, Emacs Quack (tip: open msim-api.scm and msim-extra.scm in the same Emacs session and you'll get M-/ name completion for the MSim API). The interpreter runs in a dedicated pop-up window when MSim is run as an applet or via Java Web Start.

Graphical User Interface

The GUI renders the current model at 15FPS by default; this can be changed with (set-min-render-ms minimum-frame-cycle-time-ms). When not explicitly manipulating a joint, you navigate to different viewpoints using the arrow keys and/or the mouse. Navigation works as if you were manipulating the pose of the ground link. Type 'h' or '?' in the GUI to get brief context help (type ctrl-h to dump all current keybindings) including the current keyboard and mouse navigation or manipulation actions. Typically

  • the arrow keys and/or the left (primary) mouse button rotate about axes parallel to the current view
  • the arrow keys with ctrl down and/or the right mouse button translates parallel to the current view
  • the arrow keys with shift down, the middle mouse button, and/or the mouse wheel zooms in navigation mode or rotates about an axis perpendicular to the current view in manipulation mode
  • the arrow keys and/or any mouse button with alt down rotate about an axis perpendicular to the current view in navigation mode.
though there are some additional options—including keyboard modifiers that serve in the place of mouse buttons you may not have—and also some exceptions. For example, when manipulating a one-DoF joint, the primary mouse button and/or the mouse wheel actuate the joint directly.

See the UI, InteractorDriver, and RXSInteractor Javadoc for additional details.

Picking and the Selection

MSim maintains up to one entity as the current pick and an arbitrary number of entities as the current selection (the pick is not automatically considered part of the selection). You change the pick by clicking (with the left mouse button) in the GUI. Clicking in empty space or hitting escape (except when currently in a sub-mode such as joint manipulation or creating a new joint) clears the current pick. If multiple objects are under the click point you may cycle through them with page-up/page-down and/or by clicking the right/middle mouse buttons. As you do this, MSim will temporarily alpha-mask entities that may occlude the current pick.

Typing 's' will add the current pick to the selection, and shift-s will remove it. Or, hold ctrl while picking to add/remove the picked entity to/from the selection. ctrl-escape clears the selection (except when currently in manipulation mode).

By default, both the current pick and the current selection are highlighted with an emissive appearance (the highlight hue is configurable). And picking a joint or link automatically enters manipulation mode.

There are a number of additional options and APIs related to picking and the selection, see the UI Javadoc for further details.

Widgets, Routes, Visibility, and Alpha Mask

Every topological element in the MSim model is assigned an entity class depending on its type and its position in the model. Besides the kinematic graph of links and joints, at this level we also consider widgets and geometries to be part of the model. A geometry is a collection of renderable primitives: triangles, quads, line segments, or points; a widget is a collection of geometries and corresponding appearances that control their shading. Currently widgets and geometries are only used in rendering and picking.

Each widget is parented to a particular joint or link. By default, links only carry an AxesWidget, which illustrates the canonical coordinate frame of the link. Default widgets for joints typically include RouteWidgets showing the topological connection between the adjacent links and a collection of other widgets to illustrate the mobility space and configuration of the joint, depending on the joint type.

User widgets may be added to any joint or link with (add-widget joint-or-link widget-type-or-widget [widget-name] [coordinate-frame]), abbrev aw, keybinding 'shift-w', and removed with (remove-widget joint-or-link widget-name-or-widget), abbrev rw, keybinding delete. When initiated from the GUI, new widgets are added to the currently picked link or joint. To delete a widget in the GUI you first need to pick it, and by default individual widgets are not pickable: type alt-w to toggle widget picking. A number of pre-defined parametric shapes are available as widgets, such as SphereWidget, see the Javadoc for Widget and its descendants. In addition, widgets can be imported from Java3D scene graphs, which in turn can be loaded from standard file formats such as VRML. See below for details.

MSim uses the entity class assigned to a link, joint, widget, or geometry to make decisions about how to render it and whether it should be pickable. The entity classes are organized into a hierarchy to facilitate inheritance of these settings. (dump-entity-class-settings), abbrev decs, shows the current settings; various APIs are provided to adjust them. See the EntityClass Javadoc for further details.

There are three possibilities for rendering each entity:

  • visible: rendered normally
  • hidden: not rendered
  • masked: ghost-rendered in an alpha-masked rendering pass.

Importing Models

Importing a robot model essentially consists of developing a script to construct the kinematic structure of the model, and then loading user widgets to add renderable geometry (typically to the links). See the (load-athlete) script in MSim-ATHLETE for an example.

User widgets can be loaded from a number of standard file formats, including VRML 2.0 (aka VRML 97), OBJ, and 3DS. This is currently accomplished by using Java3D loaders to construct in-memory Java3D scene graphs, and then extracting selected shapes as Widgets. Ideally, the model file will be well-structured, with useful and semantic parent-child relations and coordinate frames aligned with joint axes. In this case the (load-from-j3d ...) API can potentially infer most of the kinematic structure from the Java3D scene graph. Unfortunately, many model creation programs (in particular CAD systems) do not produce such well-structured exports.

See the J3DLoader Javadoc for further details.

Known Bugs and Limitations

Download, Dependencies, and Installation

The distribution jar comes with everything rolled into one: Java source, scheme source, html (m4) source, the makefiles and related stuff, plus precompiled versions of everything (Java class files, html, and Javadoc). Further, it includes the pure Java dependencies of MSim, which are

A lighter-weight jar is also available which includes MSim and VonaUtils but no third-party dependencies, and there is also an extra-light jar which contains MSim only.

Native Dependencies

MSim requires a Java 1.6+ JRE and JOGL 2.0+. It has been tested under recent Sun JREs on GNU/Linux, Windows XP, Mac and OS 10.6.

MSim uses netlib-java for computation of the SVD. While netlib-java will default to a pure-Java implementation, it is much faster to use a JNI-accelerated call to an optimized native SVD implementation. You will likely need to take special action to enable this.

Invoking

Once you have a compatible JRE and JOGL (and optionally the netlib-java JNI acceleration) installed you can run MSim like this:

java -jar MSim-newest.jar

(Re-)Building the Code

The MSim build has been tested on GNU/Linux. Other platforms may work. You should only need to (re)build the code if you have made a change; it is distributed with pre-built binaries.

The build system is based on the Super-Ninja Makefile. You'll need to have already installed

  • a 1.6+ JDK with the binaries on your path
  • (GNU) make and m4
  • the JOGL library.

You'll probably want to create a directory into which to unpack the jar (jar xvf MSim-newest.jar). The code for MSim proper will be under the unpacked directory msim/; sibling directories contain the bundled pure-Java dependencies. The build system will find these. Alternately, you can unpack MSim-lite-newest.jar, and in that case, the build system will look for jarfiles for the pure Java deps in the location specified by its EXT_DIR variable, which you can inspect via make show-settings (and/or see the makefile for how it is derived).

From within the msim/ directory,

  • make show-settings dumps the build system's config vars
  • make makefiles generates makefiles in any subpackages
  • make project-clean removes most precompiled stuff
  • make project-realclean removes all precompiled stuff
  • make project (re)builds the binary files for the project
  • make (re)builds the binary files in the current directory
  • make project-javadoc (re)builds the project documentation
  • make jar (re)builds the jar
  • make jar-lite (re)builds the lite jar

Roadmap

  • further solver usability refinements
  • on-line integration with hardware controls e.g. BRBrain
  • jacobian visualization
  • goal position ghosting
  • general purpose trajectory import/export
  • undo/redo
  • manipulation snapping
  • rendering optimization: display lists, view-dependent tesselation
  • planar case solve and render optimizations
  • more GUI features: labels, palettes, HUD, rectangle select
  • additional model import/export (URDF, DH, Webots, COLLADA)
  • useful applet functionality (requires non-Java3D model loader)
  • simulation of physical dynamics
  • collision detection with specified flats
  • general collision detection
  • better multicore utilization
  • perspective projection

Acknowledgements

The ATHLETE VRML model shown in some demo animations was provided courtesy of RSVP Team NASA/JPL/Caltech (that model is proprietary; an alternate simple model is included in the download).

This project was funded in part under the NASA/JPL SURP and NSF EFRI programs.

References

AS96Hashem Ashrafiuon and Kiran Sanka. Development of virtual link method for the solution of hyper-redundant spatial robots. Journal of Robotic Systems, 13, 1996.
BB04Paolo Baerlocher and Ronan Boulic. An inverse kinematics architecture enforcing an arbitrary number of strict priority levels. The Visual Computer, 20, 2004.
CMPS02C. L. Collins, J. M. McCarthy, A. Perez, and H. Su. The Structure of an Extensible Java Applet for Spatial Linkage Synthesis. Journal of Computing and Information Science in Engineering, March 2002.
CNNK06Antonio Diaz-Calderon, Issa A. D. Nesnas, Hari Das Nayar, Won S. Kim. Towards a Unified Representation of Mechanisms for Robotic Control Software. International Journal of Advanced Robotic Systems 3(1), 2006.
Flu98Fluckiger, L. A Robot Interface Using Virtual Reality and Automatic Kinematics Generator. The 29th International Symposium on Robotics, 1998.
Gra98F. Sebastian Grassia. Practical parametrization of rotations using the exponential map. J. Graph. Tools, 3(3), 1998.
HA05Christoph M. Hoffmann, Robert Joan-Arinyo. A Brief on Constraint Solving. Computer Aided Design and Applications, April, 2005.
Hof01Christoph M. Hoffmann. D-Cubed's Dimensional Constraint Manager. Journal of Computing and Information Science in Engineering Software, 2001.
IG98Oleg Ivlev and Axel Graser. Resolving redundancy of series kinematic chains through imaginary links. Proceedings of CESA IMACS Multiconference, Computational Engineering in Systems Applications, 1998.
Lie77A. Liegeois. Automatic supervisory control of the configuration and behavior of multibody mechanisms. IEEE Trans. on Systems, Man, and Cybernetics, SMC-7 (12), 1977.
PCTDP01Jerry Pratt, Chee-Meng Chew, Ann Torres, Peter Dilworth, Gill Pratt. Virtual Model Control: An Intuitive Approach for Bipedal Locomotion. The International Journal of Robotics Research, Vol. 20, No. 2, 2001.
PTKT02Mitchell W. Pryor, Ross C. Taylor, Chetan Kapoor, Delbert Tesar. Generalized Software Components for Reconfiguring Hyper-Redundant Manipulators. IEEE/ASME Transactions on Mechatronics, 7(4), December 2002.
PZB90Cary B. Phillips, Jianmin Zhao, and Norman I. Badler. Interactive real-time articulated figure manipulation using multiple kinematic constraints. SIGGRAPH Comput. Graph., 24(2), 1990.
SS91B. Siciliano and J.-J.E. Slotine. A general framework for managing multiple tasks in highly redundant robotic systems. Fifth International Conference on Advanced Robotics, 1991.
Sut63Sutherland, I. E.. Sketchpad: A Man-Machine Graphical Communication System. Ph.D. Thesis, Massachusetts Institute of Technology, 1963.
Wel93Chris Welman. Inverse kinematics and geometric constraints for articulated figure manipulation. MS thesis, Simon Fraser University, 1993.
Whi69D. E. Whitney. Resolved motion rate control of manipulators and human prostheses. IEEE Transactions on Man Machine Systems, 10(2), June 1969.

Disclaimer

THIS INFORMATION AND/OR SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS INFORMATION AND/OR SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.