{"id":1593,"date":"2018-12-13T09:40:10","date_gmt":"2018-12-13T08:40:10","guid":{"rendered":"http:\/\/blog.creamontblanc.org?p=1593"},"modified":"2018-12-13T09:40:25","modified_gmt":"2018-12-13T08:40:25","slug":"do-you-see-what-i-see-interobserver-variability-in-citizen-science","status":"publish","type":"post","link":"https:\/\/blog.creamontblanc.org?p=1535","title":{"rendered":"Do you see what I see? Interobserver variability in citizen science"},"content":{"rendered":"<p><strong>Participants in CREA Mont-Blanc\u2019s citizen science program Ph\u00e9noclim make repeat observations of tree phenology. To help us study whether or not our data is reliable, it is useful to know if two volunteer observers independently make the same evaluation of the same tree.\u00a0 Here are the results and lessons learned from the experiment we carried out with students from\u00a0<a href=\"https:\/\/www.fus.edu\/\">Franklin University Switzerland<\/a> this fall.<\/strong><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1559 size-large\" style=\"font-weight: bold\" src=\"http:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/201810_Univ_Franklin_Vallorcine_Loriaz_CVR_CREAMontBlanc-1000x500.jpg\" alt=\"\" width=\"1000\" height=\"500\" srcset=\"https:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/201810_Univ_Franklin_Vallorcine_Loriaz_CVR_CREAMontBlanc-1000x500.jpg 1000w, https:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/201810_Univ_Franklin_Vallorcine_Loriaz_CVR_CREAMontBlanc-300x150.jpg 300w, https:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/201810_Univ_Franklin_Vallorcine_Loriaz_CVR_CREAMontBlanc-768x384.jpg 768w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/p>\n<p><strong><em>Students from Franklin University Switzerland observing the phenology of larch trees \u00a9 CREA Mont-Blanc<\/em><\/strong><\/p>\n<h2>Ph\u00e9noclim and the differences between observers<\/h2>\n<p>In the citizen science program, <a href=\"http:\/\/phenoclim.org\/\">Ph\u00e9noclim<\/a>, volunteer observers participate in monitoring plant phenology to help our understanding of the impacts of climate change on living things.\u00a0 The phenological event observed depends on the season, and in the fall, Phenoclim focuses on color change in leaves and needles.\u00a0 Over the course of the season, volunteers observe different tree species including birch, beech, mountain-ash and larch, recording the date when 10% and 50% of leaves or needles change color.<\/p>\n<p>Making these observations is not as easy as it might seem, and a few key questions have arisen: is there consistency among observations made by different observers? When looking at the same tree, do volunteers note the same percentage of color change?<\/p>\n<h2>The protocol<\/h2>\n<p>To help us answer these questions, 20 students from Franklin University Switzerland headed out into the field in Vallorcine (1300 meters) and Loriaz (1900 meters) with the CREA Mont-Blanc team.\u00a0 In each of the study areas, 30 larch trees had been pre-identified.\u00a0 Students visited the two study areas and evaluated the percentage of color change in each of the 30 trees.<\/p>\n<h2>Results of their observations:<\/h2>\n<p>The interactive graphic below presents the results of the observations made by the students.\u00a0 Each point represents a single student\u2019s observation of an individual tree.\u00a0 The observations from Vallorcine (1300 meters) appear in blue, while the observations made at Loriaz (1900 meters) appear in red.<\/p>\n<p>&nbsp;<\/p>\n<style>\n       .errordiv { padding:10px; margin:10px; border: 1px solid #555555;color: #000000;background-color: #f8f8f8; width:500px; }#advanced_iframe {visibility:visible;opacity:1;vertical-align:top;}.ai-info-bottom-iframe { position: fixed; z-index: 10000; bottom:0; left: 0; margin: 0px; text-align: center; width: 100%; background-color: #ff9999; padding-left: 5px;padding-bottom: 5px; border-top: 1px solid #aaa } a.ai-bold {font-weight: bold;}#ai-layer-div-advanced_iframe p {height:100%;margin:0;padding:0}<\/style><script type=\"text\/javascript\">var ai_iframe_width_advanced_iframe = 0;var ai_iframe_height_advanced_iframe = 0;var aiOnloadScrollTop=\"true\";var aiShowDebug=false;\n\t\tif (typeof aiReadyCallbacks === 'undefined') {\n\t\t\tvar aiReadyCallbacks = [];\n\t\t} else if (!(aiReadyCallbacks instanceof Array)) {\n\t\t\tvar aiReadyCallbacks = [];\n\t\t}    function aiShowIframeId(id_iframe) { jQuery(\"#\"+id_iframe).css(\"visibility\", \"visible\");    }    function aiResizeIframeHeight(height) { aiResizeIframeHeight(height,advanced_iframe); }    function aiResizeIframeHeightId(height,width,id) {aiResizeIframeHeightById(id,height);}<\/script><iframe id=\"advanced_iframe\"  name=\"advanced_iframe\"  src=\"https:\/\/plot.ly\/~ColinVR\/3\/\"  width=\"100%\"  height=\"450\"  frameborder=\"0\"  border=\"0\"  allowtransparency=\"true\"  loading=\"lazy\"  style=\";border-width: 0px;;border: none;;width:100%;;height:450px;\" ><\/iframe><script type=\"text\/javascript\">var ifrm_advanced_iframe = document.getElementById(\"advanced_iframe\");var hiddenTabsDoneadvanced_iframe = false;\nfunction resizeCallbackadvanced_iframe() {}<\/script><script type=\"text\/javascript\"><\/script><p style=\"display:block !important; visibility:visible !important;margin: -18px 14px 0 0;padding-left: 3px;padding-top:3px;background: white; overflow: hidden; position: relative; line-height:15px;width: fit-content;\"><small style=\"display:block !important;visibility:visible !important\">powered by Advanced iFrame<\/small><\/p>\n<p>&nbsp;<\/p>\n<p>This graphic shows the variability in the observations made by different observers for each tree. For example, the percentage of color change observed in tree #1 at Loriaz was recorded between 85 and 100%, depending on the observer. Another example coming from Vallorcine, where tree #1 was recorded as having between 5 and 25% color change depending on the observer.<\/p>\n<p>Unsurprisingly, the observations made by different volunteers are rarely identical (when it is the case, the dots are superposed on one another for each tree in each zone). However, can we really say that they are <em>that<\/em> different? To respond to this question, we compared the observations of each volunteer with those of each of the other volunteers. For each observer-pair, we evaluated whether or not the difference in the percentage of color change recorded were statistically significant (pair in disagreement) or not (par in agreement).<\/p>\n<p>&nbsp;<\/p>\n<figure id=\"attachment_1578\" class=\"thumbnail wp-caption aligncenter\" style=\"width: 736px\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1578 size-full\" src=\"http:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/Obs30Melezes_Accord_PasAccord_Loriaz_Vallorcine-1.png\" alt=\"\" width=\"736\" height=\"394\" srcset=\"https:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/Obs30Melezes_Accord_PasAccord_Loriaz_Vallorcine-1.png 736w, https:\/\/blog.creamontblanc.org\/wp-content\/uploads\/2018\/12\/Obs30Melezes_Accord_PasAccord_Loriaz_Vallorcine-1-300x161.png 300w\" sizes=\"auto, (max-width: 736px) 100vw, 736px\" \/><figcaption class=\"caption wp-caption-text\"><strong><em>The percentage of pairs in agreement (\u201ccouple d\u2019observateurs en accord\u201d) or not (\u201ccouple d\u2019observateurs en d\u00e9saccord\u201d) in the two study zones (Vallorcine and Loriaz). \u00a9 CREA Mont-Blanc<\/em><\/strong><\/figcaption><\/figure>\n<p>&nbsp;<\/p>\n<p>While a few observer-pairs disagreed (11-14% of observer pairs), the majority were in agreement (86-89%).<\/p>\n<h2><strong>Why these results interest us<\/strong><\/h2>\n<p>This study demonstrated there is a high probability that two different observers will record the same observation when looking at the same tree. This is particularly reassuring when we consider the reliability of data because, in Ph\u00e9noclim, a single observer is responsible for making observations of the plants in their own study zone.<\/p>\n<p>What are potential next steps for taking this line of study further? We came up with a few study ideas, but we\u2019re always open to suggestions! One possibility would be to try the same type of test on other tree species that we study in Ph\u00e9noclim. Another would be to test observer variability for other, harder-to-observe phenological stages such as budburst in springtime.<\/p>\n<p>A big thank you to the students who participated in this experiment!<\/p>\n<p>Written by: Colin Van Reeth<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Participants in CREA Mont-Blanc\u2019s citizen science program Ph\u00e9noclim make repeat observations of tree phenology. To help us study whether or not our data is reliable, it is useful to know if two volunteer observers independently make the same evaluation of &hellip; <a href=\"https:\/\/blog.creamontblanc.org?p=1535\">Lire la suite\u00ad\u00ad<\/a><\/p>\n","protected":false},"author":7,"featured_media":1560,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"ngg_post_thumbnail":0,"footnotes":""},"categories":[194],"tags":[],"class_list":["post-1593","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-la-recherche"],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/posts\/1593","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1593"}],"version-history":[{"count":8,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/posts\/1593\/revisions"}],"predecessor-version":[{"id":1603,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/posts\/1593\/revisions\/1603"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=\/wp\/v2\/media\/1560"}],"wp:attachment":[{"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1593"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1593"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.creamontblanc.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1593"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}