My first experience with rigging a quadruped hind leg was during a non-human character rigging course I took as a sophomore. The approach presented in the class covered the techniques necessary to complete a basic hind leg. The approach did not account for IK/FK switching, pinning, stretching, or non-coplanar limbs (suffice to say it was an incomplete approach).
Over the course of several projects I eventually designed a more robust hind leg rig.
The most helpful resource I could find when designing an improved hindleg was David Otte's work on Madagascar 2 and 3 as a technical director. Specifically the following resources.
Hind Leg Auto-Rigger
Beginning Of The Project
When setting up a quadruped IK system, you need all three joints (the hip, the knee, and the ankle) to rotate when you move the IK foot control. This is our first problem.
A single-chain solver would not get us the desired result because we wouldn't be able to aim the knee at a control.
A rotate-plane solver would not work because if the ankle were above the knee, it would not give a stable output. You can test a rotate-plane solver on a 3-segment joint chain to see what I mean.
So we are left with two options, a spring solver, or some sort of procedural system. Procedural systems (at least all the ones I have encountered) do not stretch all three segments, and only approximate the result given by a spring solver.
A spring solver is only stable on a planar joint chain, so if your joint chain is not planar, twist the ankle until it is coplanar with the other three joints (the hip, the knee, and the ball).
Our second problem is that a 3 segment joint chain has multiple valid IK solutions. So we need to design a system that will allow us to adjust our solution within this range.
The method that I originally learned used a primary joint chain with a spring solver on it, and a secondary joint chain with a RP solver running from the knee to the ball. The hip joint of the secondary joint chain is a child of the hip joint of the primary joint chain. This way the secondary joint chain follows along with the primary joint chain, but the secondary hip joint is still free to rotate and offset from the solution the primary joint chain gives. The issue with this is that it prevents stretching. If the leg was stretched, you could set up the joints to scale like in a normal stretchy biped leg, but the offset on the secondary joint chain would remain constant and so the secondary joint chain would not straighten properly.
Goals For The System
Before we begin rigging the hind leg we need to figure out what the resulting rig will need to be able to do. We want the hind leg rig to be able to:
Switch between IK and FK
Stretch when in IK
Switch between a digitigrade and plantigrade mode
This will allow the system to be used on character who switch between the two, an will also allow us to "pin" the ankle to the ground like when a dog sits on it's hind legs.
Lock the the upper and lower segments of the leg so that characters who switch from quadruped to biped (like Marty the Zebra from Madagascar) can do this.
When in digitigrade mode we want the hip segment to stretch to the PV control, and the knee and ankle segments to behave like an RP handle is running from the PV control to the ball of the foot.
When the two lower segments are locked we want the hip segment to stretch to the PV control, and the knee and ankle segments to stretch uniformly so they cover the distance from the PV control to the ball of the foot. (The difference between these two is subtle, but in this setup the the knee and ankle segments may also shrink while in the digitigrade mode they will only stretch and will bend when the distance from the PV to the ball is less than the segments normal length.) Below is a visual example of what I mean by that.
When in plantigrade mode we only want the hip and knee joint to stretch to the PV control while the ankle does not scale.
Normal Digitigrade Pinning
Pinning While Locked
What we need is several valid solutions running in parallel, that we can blend between them. We will have 3 joint chains each with their own IK setup. The one in red is running a spring solver. The one in yellow is running an RP solver but the ankle joint has rotation transform limits that keep it "locked". The one in green is running an RP solver with the knee joint "locked". These represent the minimum and maximum rotations of the hip joint that would yield a valid IK solution. We will refer to these two new chains as the forward (FWD) and backward (BKW) chains.
As a bonus, these joint chains will help use "lock" the lower segments just like on Marty the Zebra.
There is one remaining issue with this system. Because the spring chains hip cannot inherit rotations it would be point constrained to a hip move control (the best way to prevent this is to turn off "inherit transforms" on the joint itself or the group above it). This means the spring hip and the other two hips are not rotating in the same space. There are two possible solutions, we can either blend between them using an orient constraint, or we can extract the rotation from the spring hip. The system we will use to blend between the chains uses actual rotate values, and wouldn't work with the 0 to 1 range of an orient constraints weight. So we will extract the rotations from the spring hip.
The first step in the node network is the hip offset. We will add the rotation from the spring solver (extracted using the locator) to a "hip offset" attribute on the IK foot control. We will then clamp that value between the rotation from the FWD and BKW chains so that the user cannot offset the hip into an invalid position.
So we need find out what the rotations of the spring hip would be if it were in the same local space as the two other chains. To do this we will create an empty transform node (a group) that is sibling of either the FWD or BKW hip joint. We will set it's rotations to match the orientation of the spring hip joint. We will then put a locator under this group. Originally I had planned to parent constrain this locator to the spring hip to extract the rotations, but I discovered that parent constraints are prone to flipping in these sorts of setups. So instead I used an aim constraint and aimed the locator at the spring knee joint using the FWD hip joint as the world up object and the axis perpendicular to the plane as the world up axis.
At this point it is important to talk about rotate order. The axis that is perpendicular to the plane the joints lie in needs to be last in the rotation order. I made x the primary axis and y the secondary axis so my rotate order is YXZ (I chose this over YZX because YXZ is less likely to produce gimble lock). Both the locator and the joints need to have this rotate order. The group above the locator should have the XYZ rotate order because Maya uses that order when handling the orientation of joints. If you don't know what I mean when I talk about orientation of joints being different from their rotations here is a video that covers that info (VIDEO).
Now that we have the rotate values of the three joint chains let's go over blending them. The joint chain we will using to blend them is identical to the one the hip offset method uses. We will be running an RP solver from the knee to the ball and plugging rotations into the hip joint. The rotations come from this node network (I have broken the node network up into several images for clarity).
But we also want to be able to lock the knee or ankle regardless of what offset value is required, so we have another attribute which I called "BKW/Spring/FWD" that ranged from -1 to 1 (-1 being blend fully to the BKW rotation , 0 being use the result of the clamp node, and 1 being blend fully to FWD rotation). There are many ways to do this but the simplest way is to use a single remap value node. Now that might seem sufficient but there is actually one more step.
When knee pinning is on we need make sure that the hip joint is aimed at the PV control, so when pinning is on we need to have the hip joint inherit the rotations of the FWD hip joint. We use a blendTwoAttr node to blend between the result o the remap value and the FWD chains rotation. We also plug in the rotate-X and rotate-Y from the FWD hip joint to make sue the planar chain which we are using to blend between the solutions remains in the correct plane.
Pole Vector Stuff
To finish of the planar chain we need to set up a pole vector control. The FWD RP ikHandle handle and the spring ikHandle will both be poleVector constrained to this control. The RP ikHandle on the planar chain and the BKW chain face need the opposite poleVector so simple multiply the output from the poleVectorConstraint node by -1 and plug that into the pole vector for these two chains.
The last thing we have to do is make the planar chain non-planar (strange I know). We will use the twist attribute on the planar chains ikHandle to twist the ankle back into it's original non-planar position.
Plantigrade Mode (Sitting)
Setting up the plantigrade mode for the hind leg is fairly simple, because it is the same as on a normal biped leg. The only difference is that you will have to have an RP ikHandle running from the ankle to the ball not an SC solver. For that reason I will not go over this part of the rig. If you are unsure how to set this part up here is a tutorial. Don't set up this system on the hind leg yet though, you just need an understanding of how a normal biped leg works.
In order to blend between the plantigrade and digitigrade modes we will blend between two sets of rotations for the locator that is a parent of the plantigrade RP ikHandle. We will blend between the rotations needed to put the ankle joint on the ground, and the rotations needed to match the planar joint chain. To get the second set of rotations we will aim a locator located at the ball at the planar chains ankle joint. The IK foot control should be the world up object for this aim constraint. We will blend between these two sets if rotations using a blendColors node.
In order for the knee of the plantigrade mode to match the knee of the digitigrade setup we need to set up a pole vector, but using the same PV control will not work because the platigrade ikHandle has a different end position. We could use the planar knee joint as the pole vector target but that would cause flipping when the joint chains was straightened because the two knee joints would be right on top of each other.
Instead we will create a locator that is a child of the planar knee joint and translate -.01 along the primary axis of the joint, and use this as the pole vector target. That way even when the two knee joints are aligned, the locator will still provide the correct pole vector.
The last challenge of blending between these two modes is twisting. The shin segment should twist gradually when in digitigrade mode, but twist along with the foot completely with the foot while in plantigrade mode. To make this happen we will blend between two poleVector targets. The first target will be a locator offset slightly from the planar ankle that is a child of the planar ankle. The other target is a locator offset from the position the ankle will be in when the character is sitting. This second locator will be a child of the
Stretching and Pinning
Now before you commit to setting up pinning on your hindleg, decide if you really need it, because it is the most complicated part of the rig.
Let hearken back to when we set some goals for the rig.
So we have 5 unique behaviors.
Pinned and locked digitigrade
These behaviors will be controlled by the pinning attribute and the "BKW/Spring/FWD" attribute. So we will go over how to set them up individually and then show how to blend them.
For stretchy digitigrade the scale of the joints is based on the following math:
length ratio = (distance from hip to ball / global scale) / Σ(starting length of segment * stretch attribute)
result scale = clamp[1,stretch limit](length ratio) * stretch attribute
Σ is a symbol meaning sum in this case it means:
starting distance from hip to knee * hip stretch attribute +
starting distance from knee to ankle * knee stretch attribute +
starting distance from ankle to ball * ankle stretch attribute = Σ(starting length of segment * stretch attribute)
Now we have three outputs (one for each joint). We will call these the hip_dig_stretch, knee_dig_stretch, and ankle_dig_stretch. (get rid of input in name of nodes)
The node editor for this setup looks like this:
The stretchy plantigrade setup is almost identical. The only difference is we measure the distance from the hip to the ankle (using a locator that is a child of the locator we used to blend between the sit and planar joint chains) instead of the distance from the hip to the ball, and we only add the scaled original length of the hip and knee segments.
From this setup we get two outputs. We will call these hip_sit_stretch, and knee_sit_stretch. In this mode there is no output for the ankle joint because it does not scale (the output would always be 1).
Here for clarity is the node network for this setup:
When in pinned plantigrade mode the hip joint will scale so the knee is located at the PV control and the knee will scale so that the ball is located at the foot ctrl. This is accomplished using the following math:
hip_sit_pin = (distance from hip to PV control / global scale) / (original distance from hip to knee)
knee_sit_pin = (distance from PV control to ankle / global scale) / (original distance from knee to ankle)
When pinning in digitigrade mode we want the knee to snap to the PV control, and the knee and ankle segments to act like they have an RP solver running along them.
The hip can reuse the output from the platigrade pinning.
The knee and ankle scale will come from a system similar to the stretchy plantigrade. This system uses the following math:
length ratio = (distance from PV to ball / global scale) / Σ(starting length of segment * stretch attribute)
result scale = clamp[1,stretch limit](length ratio) * stretch attribute
We will call this resulting scale knee_pin_unlock and ankle_pin_unlock.
Because we also have the option to pin when the two lower segments are locked together we have another node network which uses the following math:
length ratio = (distance from PV to ball / global scale) / Σ(starting length of segment * stretch attribute)
result scale = (length ratio) * stretch attribute
The only difference in the math is that the locked pinning math does not clamp the length ratio. This means that the knee and ankle can have scale less than 1 in this mode. We will call the resulting scale knee_pin_lock and ankle_pin_lock.
Blending Between These Solutions
Now that we have set up node networks to give us the desired results, we need to link those networks together so we can switch between them.
The spring chain can just get the stretchy digitigrade result directly.
hip_dig_stretch = spring_hip_jnt.scaleX = spring_hip_jnt.scaleY = spring_hip_jnt.scaleZ
knee_dig_stretch = spring_knee_jnt.scaleX = spring_knee_jnt.scaleY = spring_knee_jnt.scaleZ
ankle_dig_stretch = spring_ankle_jnt.scaleX = spring_ankle_jnt.scaleY = spring_ankle_jnt.scaleZ
(the spring chain will not work properly if you scale the joints non-uniformly).
Blend the stretchy digitigrade hip solution with the platigrade hip pin solution to get the scale X for all of the planar hip joints. The driver for these blender nodes is the pin attribute.
BLEND (hip_sit_pin, hip_dig_stretch) = FWD_hip_jnt.scaleX
BLEND (hip_sit_pin, hip_dig_stretch) = BKW_hip_jnt.scaleX
BLEND (hip_sit_pin, hip_dig_stretch) = planar_hip_jnt.scaleX
Blend the the pin lock and pin unlock solutions, and then blend that result with the stretchy digitigrade solution to get the scale X for the planar knee and ankle joints. The first blender is driven by the "BKW/Spring/FWD" attribute. But because this value ranges from -1 to 1 we need to clamp it first so it ranges from 0 to 1. The second blender is driven by the pin attribute.
BLEND(BLEND (knee_pin_lock, knee_pin_unlock), knee_dig_stretch) = planar_knee_jnt.scaleX
BLEND(BLEND (ankle_pin_lock, knee_pin_unlock), ankle_dig_stretch) = planar_knee_jnt.scaleX
Because the FWD and BKW joint chains are never "unlocked" you can just blend the dig_lock_pin result with the dig_stretch result. The blender is driven by the pin attribute.
BLEND (knee_dig_lock, knee_dig_stretch) = FWD_knee_jnt.scaleX
BLEND (ankle_dig_lock, ankle_dig_stretch) = FWD_ankle_jnt.scaleX
BLEND (knee_dig_lock, knee_dig_stretch) = BKW_knee_jnt.scaleX
BLEND (ankle_dig_lock, ankle_dig_stretch) = BKW_ankle_jnt.scaleX
For the sit hip you need to first blend the pinning and stretching solution for the digitigrade mode and the pinning and stretching solutions for the plantigrade mode, and then blend those two results together. The first blending step is driven by the pin attribute, the second one which blends the two results together is driven by the sit attribute. Note that the hip_sit_pin provides the pin result for both the digitigrade and plantigrade mode.
BLEND(BLEND (hip_dig_stretch, hip_sit_pin), BLEND(hip_sit_stretch, hip_sit_pin)) = sit_hip_jnt.scaleX
For the sit knee and ankle first we will blend the lock and unlock digitigrade pin solutions using the BKW/Spring/FWD" attribute. Then we will blend that result with the stretchy digitigrade result using the pin attribute. Then we will blend the plantigrade pin and stretchy plantigrade result together using the pin attribute. and then we will blend those two results together using the sit attribute to get the final result.
BLEND (knee_dig_lock, knee_dig_unlock) using the BKW/Spring/FWD" attribute = result A
BLEND (result A, knee_dig_stretch) using the pin attribute = result B
BLEND (knee_sit_stretch, knee_sit_pin) using the pin attribute = result C
BLEND (result B, result C) using the sit attribute = sit_knee_jnt.scaleX
BLEND (ankle_dig_lock, ankle_dig_unlock) using the BKW/Spring/FWD" attribute = result A
BLEND (result A, ankle_dig_stretch) using the pin attribute = result B
BLEND (ankle_sit_stretch, knee_sit_pin) using the pin attribute = result C
BLEND (result B, result C) using the sit attribute = sit_ankle_jnt.scaleX
we also need to plug the ankles into the scale of the locator that drives the sit chains RP ikHandle.