Tag: technologies

Demystifying the algorithm

By Clara Bird, Masters Student, OSU Department of Fisheries and Wildlife, Geospatial Ecology of Marine Megafauna Lab

Hi everyone! My name is Clara Bird and I am the newest graduate student in the GEMM lab. For my master’s thesis I will be using drone footage of gray whales to study their foraging ecology. I promise to talk about how cool gray whales in a following blog post, but for my first effort I am choosing to write about something that I have wanted to explain for a while: algorithms. As part of previous research projects, I developed a few semi-automated image analysis algorithms and I have always struggled with that jargon-filled phrase. I remember being intimidated by the term algorithm and thinking that I would never be able to develop one. So, for my first blog I thought that I would break down what goes into image analysis algorithms and demystify a term that is often thrown around but not well explained.

What is an algorithm?

The dictionary broadly defines an algorithm as “a step-by-step procedure for solving a problem or accomplishing some end” (Merriam-Webster). Imagine an algorithm as a flow chart (Fig. 1), where each step is some process that is applied to the input(s) to get the desired output. In image analysis the output is usually isolated sections of the image that represent a specific feature; for example, isolating and counting the number of penguins in an image. Algorithm development involves figuring out which processes to use in order to consistently get desired results. I have conducted image analysis previously and these processes typically involve figuring out how to find a certain cutoff value. But, before I go too far down that road, let’s break down an image and the characteristics that are important for image analysis.

Figure 1. An example of a basic algorithm flow chart. There are two inputs: variables A and B. The process is the calculation of the mean of the two variables.

What is an image?

Think of an image as a spread sheet, where each cell is a pixel and each pixel is assigned a value (Fig. 2). Each value is associated with a color and when the sheet is zoomed out and viewed as a whole, the image comes together.  In color imagery, which is also referred to as RGB, each pixel is associated with the values of the three color bands (red, green, and blue) that make up that color. In a thermal image, each pixel’s value is a temperature value. Thinking about an image as a grid of values is helpful to understand the challenge of translating the larger patterns we see into something the computer can interpret. In image analysis this process can involve using the values of the pixels themselves or the relationships between the values of neighboring pixels.

Figure 2. A diagram illustrating how pixels make up an image. Each pixel is a grid cell associated with certain values. Image Source: https://web.stanford.edu/class/cs101/image-1-introduction.html

Our brains take in the whole picture at once and we are good at identifying the objects and patterns in an image. Take Figure 3 for example: an astute human eye and brain can isolate and identify all the different markings and scars on the fluke. Yet, this process would be very time consuming. The trick to building an algorithm to conduct this work is figuring out what processes or tools are needed to get a computer to recognize what is marking and what is not. This iterative process is the algorithm development.

Figure 3. Photo ID image of a gray whale fluke.

Development

An image analysis algorithm will typically involve some sort of thresholding. Thresholds are used to classify an image into groups of pixels that represent different characteristics. A threshold could be applied to the image in Figure 3 to separate the white color of the markings on the fluke from the darker colors in the rest of the image. However, this is an oversimplification, because while it would be pretty simple to examine the pixel values of this image and pick a threshold by hand, this threshold would not be applicable to other images. If a whale in another image is a lighter color or the image is brighter, the pixel values would be different enough from those in the previous image for the threshold to inaccurately classify the image. This problem is why a lot of image analysis algorithm development involves creating parameterized processes that can calculate the appropriate threshold for each image.

One successful method used to determine thresholds in images is to first calculate the frequency of color in each image, and then apply the appropriate threshold. Fletcher et al. (2009) developed a semiautomated algorithm to detect scars in seagrass beds from aerial imagery by applying an equation to a histogram of the values in each image to calculate the threshold. A histogram is a plot of the frequency of values binned into groups (Fig. 4). Essentially, it shows how many times each value appears in an image. This information can be used to define breaks between groups of values. If the image of the fluke were transformed to a gray scale, then the values of the marking pixels would be grouped around the value for white and the other pixels would group closer to black, similar to what is shown in Figure 4. An equation can be written that takes this frequency information and calculates where the break is between the groups. Since this method calculates an individualized threshold for each image, it’s a more reliable method for image analysis. Other characteristics could also be used to further filter the image, such as shape or area.

However, that approach is not the only way to make an algorithm applicable to different images; semi-automation can also be helpful. Semi-automation involves some kind of user input. After uploading the image for analysis, the user could also provide the threshold, or the user could crop the image so that only the important components were maintained. Keeping with the fluke example, the user could crop the image so that it was only of the fluke. This would help reduce the variety of colors in the image and make it easier to distinguish between dark whale and light marking.

Figure 4. Example histogram of pixel values. Source: Moallem et al. 2012

Why algorithms are important

Algorithms are helpful because they make our lives easier. While it would be possible for an analyst to identify and digitize each individual marking from a picture of a gray whale, it would be extremely time consuming and tedious. Image analysis algorithms significantly reduce the time it takes to process imagery. A semi-automated algorithm that I developed to count penguins from still drone imagery can count all the penguins on a one km2 island in about 30 minutes, while it took me 24 long hours to count them by hand (Bird et al. in prep). Furthermore, the process can be repeated with different imagery and analysts as part of a time series without bias because the algorithm eliminates human error introduced by different analysts.

Whether it’s a simple combination of a few processes or a complex series of equations, creating an algorithm requires breaking down a task to its most basic components. Development involves translating those components step by step into an automated process, which after many trials and errors, achieves the desired result. My first algorithm project took two years of revising, improving, and countless trials and errors.  So, whether creating an algorithm or working to understand one, don’t let the jargon nor the endless trials and errors stop you. Like most things in life, the key is to have patience and take it one step at a time.

References

Bird, C. N., Johnston, D.W., Dale, J. (in prep). Automated counting of Adelie penguins (Pygoscelis adeliae) on Avian and Torgersen Island off the Western Antarctic Peninsula using Thermal and Multispectral Imagery. Manuscript in preparation

Fletcher, R. S., Pulich, W. ‡, & Hardegree, B. (2009). A Semiautomated Approach for Monitoring Landscape Changes in Texas Seagrass Beds from Aerial Photography. https://doi.org/10.2112/07-0882.1

Moallem, Payman & Razmjooy, Navid. (2012). Optimal Threshold Computing in Automatic Image Thresholding using Adaptive Particle Swarm Optimization. Journal of Applied Research and Technology. 703.

How to apply my PhD?

By Leila Lemos, PhD candidate, Fisheries and Wildlife Department

Time has flown. It seems that it was like a month ago that I received the news that I was approved in a public notice from the Brazilian government to study abroad, and began the process of moving to Oregon. But actually almost three years have now passed, and I am starting to wrap up my PhD, since I need to defend it in a little bit more than a year.

Our team is now starting the third and last fieldwork season for my PhD project. I am also working on my study plan to determine the last classes I need to take, and our first manuscripts are ‘in press’ or ‘in prep’ for submission to journals. So, it’s time for me to think about what comes next.

I am from Rio de Janeiro, Brazil, and I am studying in the US through a Brazilian government program called Science Without Borders. This program aims to send students abroad to learn new techniques and to develop innovative projects. The projects needed to be original to be approved by the public notice. The main idea is to bring these students back to Brazil, after their PhD completion, to disseminate the acquired knowledge by applying the learned techniques.

My project includes a few novel aspects that allowed for funding by this program. The main focus of my thesis is to develop an endocrinology study of a cetacean species. This was (and still is) a critical field in Brazil, as reported by the “National Action Plan for the conservation of aquatic mammals: Small cetaceans” (2010). According to this Action Plan, cetacean hormonal analyses are rare and of high priority, but there are limited labs with the capacity to study cetacean endocrinology in Brazil. Other limiting factors are the associated analysis costs and a lack of human knowledge and skills. In addition to the hormonal analyses (Figure 1), I am also using other ‘new technologies’ in the project: drones (Figure 2; Video 1) and GoPros (Video 2).

Figure 1: Learning how to perform hormonal analysis at the Seattle Aquarium, WA.
Source: Angela Smith

 

Figure 2: Learning how to fly a drone in Newport, OR.
Source: Florence Sullivan

 

Video 1: Drone flights performed in Newport, OR, during fieldwork in 2016.

* Taken under NOAA/NMFS permit #16111 to John Calambokidis.

 

Video 2: Video of mysid swarms during a GoPro deployment conducted in Port Orford, OR, during fieldwork in 2016.

 

The importance of studying cetacean hormones includes a better understanding of their reproductive cycles (i.e., sex hormones such as progesterone, testosterone and estradiol) and their physiological stress response (i.e., cortisol) to possible threats (e.g., acoustic pollution, contaminants, lack of prey). In addition, through photographs and videos recorded by drones we can conduct photogrammetry analysis to monitoring cetacean body condition, and through GoPro recordings of the water column we can assess prey availability. Changes in both body condition and prey can help us explaining how and why hormone levels vary.

Through my PhD I have obtained skills in hormone analysis, photogrammetry and video prey assessment by studying the logistically accessible and non-threatened gray whale (Eschrichtius robustus). During method development, these features are important to increase sample size and demonstrate feasibility. But now that the methodologies have proven successful, we can start applying them to other species and regions, and under different circumstances, to improve conservation efforts of threatened populations.

Many cetacean species along the Brazilian coast are threatened, particularly from fishing gear and vessel interactions, chemical and noise pollution. By applying the methods we have developed in the GEMM Lab during my PhD to cetacean conservation issues in Brazil, we could enable a great expansion in knowledge across many fields (i.e., endocrinology, behavior, photogrammetry, diet). Additionally, these skills can promote safer work environments (for the scientist and for the object of study) and cheaper work processes. However, many countries, such as Brazil, do not have the infrastructure and access to technologies to conduct these same analyses, as in developed countries like the USA. These technologies, when sold in Brazil, have many taxes on the top of the product that they can become an extra hurdle, due to budget constraints. Thus, there is a need for researchers to adapt these skills and technologies, in the best manner possible, to the reality of the country.

Now that I am starting to think about ‘life after PhD’, I can see myself returning to my country to spread the knowledge, technologies and skills I have gained through these years at OSU to new research projects so that I am able to assist with conservation efforts for the ocean and marine fauna in Brazil.

 

References:

PAN, 2010. Plano de ação nacional para a conservação dos mamíferos aquáticos: pequenos cetáceos / André Silva Barreto … [et al.]; organizadores Claudia Cavalcante Rocha-Campos, Ibsen de Gusmão Câmara, Dan Jacobs Pretto. – Brasília: Instituto Chico Mendes de Conservação da Biodiversidade, Icmbio, 132 p. Em: <http://www.icmbio.gov.br/portal/images/ stories/docs-plano-de-acao/pan-peqs-cetaceos/pan_pequenoscetaceos_web.pdf> Acessado em: 27 de Maio de 2015.