Machine-Learnt Physically Based Animation (Shelved)

See the code for the agent here:

I'm working on active ragdoll animation with machine learning. The goal is for the dog character to shift its movements based on the player's directions, whether it's changing speeds or directions, or even switching between different walking patterns. I realise this is a challenging project, and potentially not feasible given the time constraints and level of complexity, but the progress so far is promising.

I've decided to move on from the dog iteration of this project, and take it in another direction. Whilst I achieved some success with this attempt, I think the particular issue I was trying to solve was a bit too much of a stretch. Maybe this is something I can come back to if I manage to develop my knowledge on this topic whilst working on a variant that is slightly more doable. I have listed some of the issues below, along with a few other things I wanted to change for the next iteration.

• I had been using quite a tall creature to animate (a fairly lanky dog), relative to its width and length, this made it naturally less stable, complicating the balancing problem.
• To make things exponentially easier for the neural network I opted for using an IK system to calculate the joint angles, where the neural network decides the feet target positions only. Whilst this is a good idea from an neural net efficiency and ease of training point of view, ultimately it means the neural net has less control over how the legs move, only being able to move within the confines of the IK system. This could presumably be partially mitigated by allowing the neural net to modify the IK system in particular ways as it sees fit, however this is something I never delved further into. Ultimately, the range of movements available were perhaps too limited for this particular problem.
• To get a creature to be able to move to a target location without falling over is one thing, to get it to do it in a way that matches how that creature would move in the real world is an additional very difficult problem. To solve this issue in this context, you'd pretty much need to emulate the problem domain of the real world situation, this way the neural net is more likely to devise a solution which is closest to the real world solution. In the case of the dog, it would mean effectively and accurately simulating the physical properties of the dogs anatomy, i.e. resistance / damping / springiness / strength values of each joint at different positions and angles, etc. This is not something that can be achieved quickly or easily. To make things a bit easier it made sense to work on a fictional character, so if it moves in a bit of a funky way, it might look a little odd, but it wouldn't break the expectations of someone viewing it.
• Having reached a bit of a plateau of progress with the dog (not really being able to get it to achieve more than one way of moving) and knowing that my neural network knowledge was a little bit lacking, I figured this was an area I could do more research on to help me achieve better results.

Building on what I'd learnt from the previous iteration, I made the following changes:

• I made the character closer to the ground relative to its height, this makes it easier to balance its self.
• I made the character fictional, so any potential odd movements wouldn't be immersion breaking. This also removes the need to painstakingly emulate a real world locomotion problem.
• Before the previous iteration, I had completed a basic racing game using the same machine learning package. This particular project was a comparatively simple implementation of machine learning, where the neural network had only 50 observations (inputs), a few layers in between, and 4 actions (outputs). My limited knowledge on machine learning was enough to get this project over the line, however I felt that delving deeper would be beneficial for this more complicated project, where there is a lot more going on. For this project, not only are there a significantly greater number of observations (approx. 300) and actions (approx. 50), but the problem its self is substantially more complex. Ultimately, a lot of neural network tuning is based on very time consuming trial and error, it's incredibly difficult to know for sure how to set the training algorithm settings and network structure for any given problem. This being said, after spending a good while delving deep into the implications of different hyperparameters and network settings, I made a number for changes and this had a very positive impact on the competency I was able to achieve with this particular iteration.

The end result (as of now) for this project, is I have managed to create a creature which can effectively walk and climb. To achieve these results, I overcame a number of different issues, further developed my understanding of how the training algorithm works, further developed the training manager, further developed the reward structure, and created an environment which automatically increases in difficulty based on the competency of the model throughout training.

I had been hoping to get the creature to be able to jump, and whilst it definitely gives it a go, and sometimes manages to do it, its far from any acceptable level of competency.

I have a theory as to why this might not yet be possible, this is as follows. Velocity and angular velocity readings in Unity are not 100% accurate, especially where higher force collisions are involved (such as pushing off for / landing a jump). As these readings are part of how the agent 'sees' the world, if the readings are not accurate, the actions the agent takes based off of these observations will not be appropriate. I feel like accuracy of these inputs are not overly important for walking and climbing (so long as they are near enough). This is partially due to the fact that the agent has the ability to 'grab', and so if actions forces are a little out it's not the end of the world, it can still keep itself oriented, the story for jumping is slightly different. Additionally, physics in Unity is not fully deterministic, so even if it did have fully accurate information, it's not certain that the actions it takes would necessarily lead to the results it expected.

A potential way to improve this situation could be to calculate the forces involved between the leg and the ground during the jump, and feed these into the neural network, so it has a better picture of what's going on. As I'm using articulation bodies, rather than rigidbodies, this might not be too difficult to achieve. It might also be a good idea to feed the forces that are acting on the joints to the neural network, as these are readily available when using articulation bodies.

As of now, I am conscious of how long this project is taking me. Each change I make might only take a few hours to implement, but then there is at least another 10 hours of training on top to evaluate the impact of the changes. It's usually wise to keep an eye on the training, this can help with identifying why things go wrong when they do, so effective fixes can be made. Training neural nets is messy and complicated, often the smallest of changes can have massive unexpected impacts. Whilst I'm happy with the current state of this project, and I see potential to improve the competency further to the point where it's viable, I am going to be shelving this project for now. This will free up the time to work on something which is a lot more doable in a shorter time period. I'd like to have a go at putting a full, but technically simple game out, without the constraints of uni deadlines. I feel like this is something that would greatly improve my portfolio.