How do I solve this problem

That expression is really a mess.

                        5x³ + 4x² + x³

First of all, notice that the first and third terms are amounts of the
same thing ... you have  5 of x-cubed, and then another 1 of x-cubed. 
There's no reason they have to stay separate like that. 

5 of anything plus another 1 of the same thing is always 6 of them. 
This is called "combining like terms".        

                          6x³ + 4x²

Now, you could factor  2  out of both terms:        2 (3x³ + 2x²)

Finally, you could factor  x²  out of both terms:        2x²(3x + 2) .

I'm not sure this is any simpler than the original expression was.
But I think this is what the homework exercise is looking for.

Aand b is what i think