Abag contains 3 red marbles, 1 green one, 1 lavender one, 3 yellows, and 2 orange marbles. how many sets of five marbles include at most one of the yellow ones?

18 marbles altogether. 

Lavender without any yellow: 
1 way to choose the lavender 
14 others that aren't yellow. 
14C4 ways to choose 4 others. So one answer is 1 X 14C4 = 14C4 = 1001, but that's only if the marbles of the same color are distinguishable. 
Since you normally can't tell one colored marble from another, it'll take more work. E.g., labeling the marbles by the initial of their color, and assigning numbers to them, you could have 
1L, 1R, 2R, 3R and 1G 
or 
1L, 1R, 2R, 3R and 2G, and I think you want to consider those the same "set," since one Green marble is as good as another. If that's not the case, then it doesn't make any difference how many marbles there are that aren't yellow. 

So you need to find how many ways you can make up a set of 4 marbles to add to 1L, without any Yellows. 
You can have 
3 Red + 1 other (G or O) -- that makes two ways 
2 Red + 2 others (GG, OO or GO) -- that makes 3 ways 
1 Red + 3 others (GGG, GGO, GOO, or OOO) -- 4 ways. 
That's all the ways with any Red marbles, a total of 2+3+4=9 

Now let's count the ways with Green ones, but no red 
 
GGG + O 
GG+ OO 
G + OOO 
O+OOO 
That's 5 more ways, for a total of 9+5 = 14 ways 

Now switching to 1 Yellow and no Lavender, you'll have exactly the same 14 ways of adding 4 non-Lavender marbles to the 1 Yellow. 

So the total is 14+14=28 ways. 

Complicated combinatorial problems like this just have to be broken down into all the cases. There may be a convenient formula that can be used from time to time, but in general there's no formula that's going to help in a non-standard problem like this. You almost always have to start, with, "Assume there are 0, 1, 2, some max) number of one item..." and go from there. 

The other thing you need to pay attention to is what objects are distinguishable and what aren't, as noted above where 1001 is the (easy to get) answer if the marbles are all different.