Sfrgttk

How many serial ports are on the Pi 3?

Central Vacuuming: Is it worth it, and how does it compare to normal vacuuming?

How does the math work when buying airline miles?

Is there hard evidence that the grant peer review system performs significantly better than random?

How could we fake a moon landing now?

What is the difference between globalisation and imperialism?

Should there be a hyphen in the construction "IT affin"?

How to plot logistic regression decision boundary?

Lagrange four-squares theorem --- deterministic complexity

How were pictures turned from film to a big picture in a picture frame before digital scanning?

"Lost his faith in humanity in the trenches of Verdun" — last line of an SF story

How much damage would a cupful of neutron star matter do to the Earth?

Strange behavior of Object.defineProperty() in JavaScript

Why are vacuum tubes still used in amateur radios?

A term for a woman complaining about things/begging in a cute/childish way

An adverb for when you're not exaggerating

What is the meaning of 'breadth' in breadth first search?

Importance of からだ in this sentence

What would you call this weird metallic apparatus that allows you to lift people?

How many time has Arya actually used Needle?

What do you call the main part of a joke?

Crossing US/Canada Border for less than 24 hours

Time to Settle Down!

How to get all distinct words within a set of lines?

Interpretation of R output from Cohen's Kappa

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

1

$begingroup$

I have the following result from carrying out Cohen's kappa in R

library(irr)

n = 100

o = c(rep(0,n), rep(1,n))

p = c(rbinom(n,1,0.5), rbinom(n,1,0.51))

k = kappa2(

  data.frame(p,o), "unweighted"

)

k

Which outputs

 Cohen's Kappa for 2 Raters (Weights: unweighted)



 Subjects = 200 

   Raters = 2 

    Kappa = -0.08 



        z = -1.13 

  p-value = 0.258 

My interpretation of this is that the test is displaying that there seems to be disagreement between the two vectors as kappa is negative. However, given the p value of 0.258 we can't say that this disagreement is significant, and may just be down to chance.

If
someone could highlight if there is anything I'm missing from this interpretation that would be appreciated.

hypothesis-testing model-comparison agreement-statistics association-measure cohens-kappa

share|cite|improve this question

asked 43 mins ago

baxxbaxx

293111

$endgroup$

add a comment | 

1

$begingroup$

I have the following result from carrying out Cohen's kappa in R

library(irr)

n = 100

o = c(rep(0,n), rep(1,n))

p = c(rbinom(n,1,0.5), rbinom(n,1,0.51))

k = kappa2(

  data.frame(p,o), "unweighted"

)

k

Which outputs

 Cohen's Kappa for 2 Raters (Weights: unweighted)



 Subjects = 200 

   Raters = 2 

    Kappa = -0.08 



        z = -1.13 

  p-value = 0.258 

My interpretation of this is that the test is displaying that there seems to be disagreement between the two vectors as kappa is negative. However, given the p value of 0.258 we can't say that this disagreement is significant, and may just be down to chance.

If
someone could highlight if there is anything I'm missing from this interpretation that would be appreciated.

hypothesis-testing model-comparison agreement-statistics association-measure cohens-kappa

share|cite|improve this question

asked 43 mins ago

baxxbaxx

293111

$endgroup$

add a comment | 

$begingroup$

I have the following result from carrying out Cohen's kappa in R

library(irr)

n = 100

o = c(rep(0,n), rep(1,n))

p = c(rbinom(n,1,0.5), rbinom(n,1,0.51))

k = kappa2(

  data.frame(p,o), "unweighted"

)

k

Which outputs

 Cohen's Kappa for 2 Raters (Weights: unweighted)



 Subjects = 200 

   Raters = 2 

    Kappa = -0.08 



        z = -1.13 

  p-value = 0.258 

My interpretation of this is that the test is displaying that there seems to be disagreement between the two vectors as kappa is negative. However, given the p value of 0.258 we can't say that this disagreement is significant, and may just be down to chance.

If
someone could highlight if there is anything I'm missing from this interpretation that would be appreciated.

hypothesis-testing model-comparison agreement-statistics association-measure cohens-kappa

share|cite|improve this question

asked 43 mins ago

baxxbaxx

293111

$endgroup$

library(irr)

n = 100

o = c(rep(0,n), rep(1,n))

p = c(rbinom(n,1,0.5), rbinom(n,1,0.51))

k = kappa2(

  data.frame(p,o), "unweighted"

)

k

 Cohen's Kappa for 2 Raters (Weights: unweighted)



 Subjects = 200 

   Raters = 2 

    Kappa = -0.08 



        z = -1.13 

  p-value = 0.258 

share|cite|improve this question

asked 43 mins ago

baxxbaxx

293111

share|cite|improve this question

asked 43 mins ago

baxxbaxx

293111

asked 43 mins ago

baxxbaxx

293111

asked 43 mins ago

baxxbaxx

293111

1 Answer
1

active

oldest

votes

2

$begingroup$

From the perspective of an applied analyst:

First note: that disagreement means if rater A says 1 rater B says 0; it is like how a Pearson correlation of -1 denotes a strong, albeit negative, relationship. The actual null hypothesis here is: what rater A says has no relation to what rater B says.

I wouldn't make such vague yet absolute declarations such as "there seems to be disagreement" (or rather there seems to be no agreement). It is not really an appropriate summary of data without significant background and context. If we had that background and context (such as in a discussions section), we could contribute some nuanced synthesis of the result, pointing to improvements or reasons for disagreement, etc.

To interpret the results: report the percentage agreement, note if any one category was more prevalent (a case when % agreement may be high but $kappa$ may be low). State the kappa statistic and it's confidence interval. I often question the worth of a p-value where the null hypothesis is a stupid case of "no agreement", but you can quote the p-value and say that the data did not provide evidence that the raters agree.

share|cite|improve this answer

answered 21 mins ago

AdamOAdamO

35.1k264142

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f403970%2finterpretation-of-r-output-from-cohens-kappa%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

2

$begingroup$

From the perspective of an applied analyst:

First note: that disagreement means if rater A says 1 rater B says 0; it is like how a Pearson correlation of -1 denotes a strong, albeit negative, relationship. The actual null hypothesis here is: what rater A says has no relation to what rater B says.

I wouldn't make such vague yet absolute declarations such as "there seems to be disagreement" (or rather there seems to be no agreement). It is not really an appropriate summary of data without significant background and context. If we had that background and context (such as in a discussions section), we could contribute some nuanced synthesis of the result, pointing to improvements or reasons for disagreement, etc.

To interpret the results: report the percentage agreement, note if any one category was more prevalent (a case when % agreement may be high but $kappa$ may be low). State the kappa statistic and it's confidence interval. I often question the worth of a p-value where the null hypothesis is a stupid case of "no agreement", but you can quote the p-value and say that the data did not provide evidence that the raters agree.

share|cite|improve this answer

answered 21 mins ago

AdamOAdamO

35.1k264142

$endgroup$

add a comment | 

2

$begingroup$

From the perspective of an applied analyst:

First note: that disagreement means if rater A says 1 rater B says 0; it is like how a Pearson correlation of -1 denotes a strong, albeit negative, relationship. The actual null hypothesis here is: what rater A says has no relation to what rater B says.

I wouldn't make such vague yet absolute declarations such as "there seems to be disagreement" (or rather there seems to be no agreement). It is not really an appropriate summary of data without significant background and context. If we had that background and context (such as in a discussions section), we could contribute some nuanced synthesis of the result, pointing to improvements or reasons for disagreement, etc.

To interpret the results: report the percentage agreement, note if any one category was more prevalent (a case when % agreement may be high but $kappa$ may be low). State the kappa statistic and it's confidence interval. I often question the worth of a p-value where the null hypothesis is a stupid case of "no agreement", but you can quote the p-value and say that the data did not provide evidence that the raters agree.

share|cite|improve this answer

answered 21 mins ago

AdamOAdamO

35.1k264142

$endgroup$

add a comment | 

$begingroup$

From the perspective of an applied analyst:

First note: that disagreement means if rater A says 1 rater B says 0; it is like how a Pearson correlation of -1 denotes a strong, albeit negative, relationship. The actual null hypothesis here is: what rater A says has no relation to what rater B says.

I wouldn't make such vague yet absolute declarations such as "there seems to be disagreement" (or rather there seems to be no agreement). It is not really an appropriate summary of data without significant background and context. If we had that background and context (such as in a discussions section), we could contribute some nuanced synthesis of the result, pointing to improvements or reasons for disagreement, etc.

To interpret the results: report the percentage agreement, note if any one category was more prevalent (a case when % agreement may be high but $kappa$ may be low). State the kappa statistic and it's confidence interval. I often question the worth of a p-value where the null hypothesis is a stupid case of "no agreement", but you can quote the p-value and say that the data did not provide evidence that the raters agree.

share|cite|improve this answer

answered 21 mins ago

AdamOAdamO

35.1k264142

$endgroup$

share|cite|improve this answer

answered 21 mins ago

AdamOAdamO

35.1k264142

answered 21 mins ago

AdamOAdamO

35.1k264142

answered 21 mins ago

AdamOAdamO

35.1k264142