I'm working on a simplified RPG system that uses only D6s, and I want a mechanic for fumbles/critical fails.

Depending on how good the player character is, they have 1-5 dice to roll and they have to beat a difficulty set by the DM. I thought it would be fun to have players fail if they roll all 1s, but realized it makes it way too hard to fail if you have 5 dice, and a bit too easy if you have 1. Is there a more linear way of defining critical fails?

This is what I get if fumbles are on all dice showing 1s:

$begin{array}{|c|c|}
hline
textbf{Number of Dice} & textbf{Probability of Fumble} \
hline
text{1} & text{16.67%} \
text{2} & text{2.78%} \
text{3} & text{0.46%} \
text{4} & text{0.08%} \
text{5} & text{0.01%} \
hline
end{array}
$

What I would like (approximately, exact numbers are not that important):

$begin{array}{|c|c|}
hline
textbf{Number of Dice} & textbf{Probability of Fumble} \
hline
text{1} & text{18%} \
text{2} & text{15%} \
text{3} & text{12%} \
text{4} & text{9%} \
text{5} & text{6%} \
hline
end{array}
$

A close approximation to the percentages you want would use something like this:

$begin{array}{|c|c|c|}
hline
textbf{Dice} & textbf{Fumble Range} & textbf{Probability} \
hline
text{1} & text{1} & text{1/6 (16.7%)} \
text{2} & text{2-4} & text{6/36 (16.7%)} \
text{3*} & text{3-7} & text{35/216 (16.2%)} \
& text{3-6} & text{20/216 (9.3%)} \
text{4} & text{4-9} & text{126/1296 (9%)} \
text{5} & text{5-11} & text{457/7776 (5.9%)} \
hline
end{array}
$

* (3 dice could go either way)

In terms of gameplay, simpler rules are frequently better than strictly matching the desired probability distribution. I might suggest something like $N$ dice fumble on a result $le 2times N$, with a special case that a single die only fumbles on a 1 (unless you want a 1/3 chance of a fumble in the 1d case). That would give you something like:

$begin{array}{|c|c|c|}
hline
textbf{Dice} & textbf{Fumble Range} & textbf{Probability} \
hline
text{1} & text{1} & text{1/6 (16.7%)} \
text{2} & text{2-4} & text{6/36 (16.7%)} \
text{3} & text{3-6} & text{20/216 (9.3%)} \
text{4} & text{4-8} & text{70/1296 (5.4%)} \
text{5} & text{5-10} & text{252/7776 (3.2%)} \
hline
end{array}
$

I couldn't match your targets exactly, but here's a different approach

Linearity is difficult to model with only d6s as you desire, but there is a very simple method to describe that closely resembles the approximations you posit:

A character fumbles if more 1s are showing than 6s.

The resulting probabilities are:

begin{array}{rl}
text{Number of Dice} & text{Probability} \
hline
1 & 16.7% \
2 & 13.9% \
3 & 10.2% \
4 & 6.6% \
5 & 4.6% \
end{array}

Fumble if exactly one die shows a 1.

N dice are rolled on the table. If exactly one shows a 1, then it's a fumble.

begin{array}{rl}
N & P(text{fumble}) \
hline
1 & 16.67% \
2 & 13.89% \
3 & 11.57% \
4 & 9.65% \
5 & 8.04% \
end{array}

Percentile dice are an example of linear probability with more than one die. The first d10 represents tens and the second d10 represents ones so you get a linear result, from 1-100. You could do something similar with other dice, but the end result would be non-trivial to understand. For example, you could use 2d4, where the first d4 is the tens and the second d4 is the ones, but since you don't have 5-10, you can only get results of

11

12

13

14

21

22

23

24

31

32

33

34

41

42

43

44

But, within the possible outcomes, the probability is equal, or linear.

A better, more useable version is to have one die, like say a d6, and on 1-3 equals 0, and on 4-6 equals the maximum of the next die, like say 12. Then you could produce 1-24 linearly with two dice.

More intuitively, if the maximum number is greater than ten, make the first die a 10, then the second die determines whether you add to it. Like for 1-17, roll a d10 for ones, and then roll a d-whatever, where half or less is 0 and over half is +7.