Three-dimensional segmentation of computed tomography data using Drishti Paint: new tools and developments

Computed tomography (CT) has become very widely used in scientific and medical research and industry for its non-destructive and high-resolution means of detecting internal structure. Three-dimensional segmentation of computed tomography data sheds light on internal features of target objects. Three-dimensional segmentation of CT data is supported by various well-established software programs, but the powerful functionalities and capabilities of open-source software have not been fully revealed. Here, we present a new release of the open-source volume exploration, rendering and three-dimensional segmentation software, Drishti v. 2.7. We introduce a new tool for thresholding volume data (i.e. gradient thresholding) and a protocol for performing three-dimensional segmentation using the 3D Freeform Painter tool. These new tools and workflow enable more accurate and precise digital reconstruction, three-dimensional modelling and three-dimensional printing results. We use scan data of a fossil fish as a case study, but our procedure is widely applicable in biological, medical and industrial research.

YH, 0000-0002-4794-0935; JL, 0000-0002-5791-4749 Computed tomography (CT) has become very widely used in scientific and medical research and industry for its nondestructive and high-resolution means of detecting internal structure. Three-dimensional segmentation of computed tomography data sheds light on internal features of target objects. Three-dimensional segmentation of CT data is supported by various well-established software programs, but the powerful functionalities and capabilities of open-source software have not been fully revealed. Here, we present a new release of the open-source volume exploration, rendering and three-dimensional segmentation software, Drishti v. 2.7. We introduce a new tool for thresholding volume data (i.e. gradient thresholding) and a protocol for performing three-dimensional segmentation using the 3D Freeform Painter tool. These new tools and workflow enable more accurate and precise digital reconstruction, three-dimensional modelling and threedimensional printing results. We use scan data of a fossil fish as a case study, but our procedure is widely applicable in biological, medical and industrial research. synchrotron radiation phase-contrast imaging, three-dimensional scientific visualization has become more and more crucial for researchers to obtain better insights [6][7][8][9]. For over three decades, the field of three-dimensional scientific visualization has been developed considerably with many new techniques to visualize and analyse data more accurately [10][11][12][13][14][15][16][17][18][19]. Two techniques have emerged to provide different visualizations: surface rendering, the method of interpreting datasets by generating a set of polygons that represent surfaces of the desired feature; and volume rendering [10], for the reconstruction of three-dimensional structures both internally and externally. Volume rendering represents three-dimensional objects as a collection of cube-like building blocks called voxels, or volume elements. A range of well-established commercial software, such as Mimics, VG Studio and AVIZO provide numerous functionalities and good rendering outputs. However, the potential of open-source software, which is both freely available and easy to access, has not been fully explored. Three-dimensional segmentation, segmenting the internal region of interest in sequences of images, is a vital tool for investigating and understanding the internal structures of target objects [6,[20][21][22]. Two-dimensional segmentation uses each image or slice in a volumetric dataset to construct a threedimensional presentation. By contrast, three-dimensional segmenting methods, the main focus of this paper, use thresholding, edge detection, clustering or region growing techniques to group pixels based on brightness, colour or texture [21], and then render them as discrete objects.
Among 10 well-known cross-platform software programs [23], five use both surface and volume rendering, but only Drishti [24] with 'an intuitive user interface' [23], can perform three-dimensional segmentation directly on a volume (using the new tool presented here, 3D Freeform Painter). Drishti uses direct volume rendering with voxel ray casting and texture slicing algorithms; combinations of local and global illuminations along with a two-dimensional transfer function which merges colour and opacity to volume to provide realism on different materials and textures through lighting, shading or shadowing via different user-generated light volumes and change of opacity. As an open-source volume exploration and rendering software [24], Drishti has demonstrated three-dimensional rendering of high quality [6,[25][26][27].
Here, we present the most recent release of Drishti v. 2.7, and introduce new tools for three-dimensional segmentation (two-and three-dimensional painters) of volumetric data. We also introduce a new toolgradient threshold and suggest protocols for how to perform three-dimensional segmentation using Drishti Paint v. 2.7 efficiently and precisely, using the CT scan data of a fossil fish as a case study. New features in Drishti, such as mesh generation and simplification, are also explained and discussed.
Volume rendering does require datasets to be loaded into graphics memory. The minimum requirements to run Drishti are graphics processing unit (GPU)-a graphics card with 1 GB memoryand random access memory (RAM)-4GB. Drishti does not have a specific requirement for the central processing unit (CPU). For readers who do not have much computing power, we suggest trying other open-source software such as SPIERS [28] and 3D Slicer [29,30]. Both software perform excellent surface rendering and allow meshes to be handled, which can then be modified in Meshlab [31] or Blender [32].

Materials
The CT data used for this study is from a 400-million-year-old fossil fish from Burrinjuck, near Canberra, southeastern Australia (ANU V244, held at the Department of Applied Mathematics, Research School of Physics, Australian National University, Canberra). The whole specimen was scanned in 2011 and rescanned in 2015 at CT Lab, ANU [33]. The voxel size of the second scan is 21 µm.
Mango can perform three-dimensional tomographic reconstruction on CPU or GPU clusters. The CPU-only code is limited to analytic methods: Feldkamp-Davis-Kress (FDK) filtered back-projection (FBP) for a circular X-ray source trajectory and Katsevich FBP for helical/double-helical X-ray source trajectories [34]. The GPU code can also perform multi-grid iterative reconstruction [35] for a space-filling X-ray source trajectory [1]. The reconstruction code has software for automatic geometric alignment capabilities [36], X-ray source movement correction [37] and component or rigid-body sample movement correction [38].
The original whole specimen CT dataset was saved as 16-bit images. These data were loaded into Drishti Import v. 2.7. The contrast was incremented with the help of histograms, and slices were filtered by selecting the best range on the tomogram. The raw data of the whole specimen was then cropped to focus on a royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 7: 201033 selected region for the right cheek complex, which was segmented out and is used as a case study here. Raw data of the selected region of interest was saved in the format Ã .pvl.nc (i.e. processed volume format) and can be downloaded from figshare: https://doi.org/10.6084/m9.figshare.12073809. We developed a general protocol to segment the three-dimensional volumetric data using Drishti Paint v. 2.7 (see electronic supplementary material for the detailed segmentation procedure).

Results and discussion
Here, we highlight the workflow and key features of the Paint module for Drishti and the general applications of Drishti. We have developed a protocol and a workflow, summarized in figure 1, to facilitate a more efficient and accessible (i.e. open-source freeware) means of overcoming the timeintensive nature of preparing fossils digitally and extracting complex information from samples. Detailed information on new tools-gradient thresholding and 3D Freeform Painter are provided in the electronic supplementary material. Our protocol and workflow (figure 1) can be used as a baseline or starting point to be adapted to obtain an ideal result for any given dataset. The electronic supplementary material also provides details on other aspects of Drishti, including installation instructions, a summary of all Drishti-supported import formats, additional learning resources and other background information.
Three modules, Drishti Import, Drishti Render and Drishti Paint, have different capacities and features, which can be combined to ensure an accurate and precise segmentation along with volume rendering to help visualize a region-of-interest for detailed analysis. Drishti Paint uses a variety of discontinuity detectionbased and similarity detection-based image segmentation approaches. These two approaches are both implemented in two modes in Drishti Paint v. 2.7, i.e. 'Graph Cut' and 'Curve' (previously known as livewire).
The right cheek complex has been segmented from the original CT dataset for our case study. Figure 2 shows an overview of our case study, where a fossil fish skull ( figure 2a,b) is cropped (figure 2c) and segmented using the latest tools in Drishti v. 2.7 (figure 2d-e). Processed data is exported as a separate volume by using the tagging function to extract the region of interest. Segmentation (in this cheek complex example) was carried out in 16-bits full resolution in alignment with the raw data to include all information from the original scan (8-bits resolution could be used, if computing power is limited).
Two transformations in mathematical morphology (i.e. dilation and erosion) can be implemented in Drishti Paint. Dilation adds pixels to the boundaries of objects in an image, while erosion removes pixels on object boundaries. Morphological dilation makes objects more visible and fills in small holes in objects. Morphological erosion removes islands and small objects so that only substantive objects remain. These two transformations can be used in any order. We used a combination of these two  transformations with Graph Cut and 3D Freeform Painter. These two transformations are more useful when using clean and high-contrast data as there is a risk of changing the morphology of the input datasets by doing multiple transformations. We recommend keeping a copy of the original volume data, before doing multiple transformations, to avoid losing any information of the original dataset. Interpolation is another tool which can be used in Curve mode. However, interpolation is not recommended for palaeontological datasets (due to lack of reproducibility). Three types of gradient thresholding are developed and implemented in Drishti Paint v. 2.7. Values thresholding was developed in Drishti v. 2.3.1, but this is the first implementation of gradient thresholding in an open-source volume rendering software program. Gradient thresholding can be combined with values thresholding to clarify and more precisely identify the boundaries between different phases, which then makes the three-dimensional segmentation process easier. Multiplethresholding (i.e. use both values and gradient thresholding) in Drishti Paint v. 2.7 is beneficial for volume segmentation and usually is the first step towards segmenting a volume (figure 2d).
In the gradient thresholding tool, gradient type 1 uses the magnitude of central difference (i.e. surface gradient vector). Gradient type 2 takes the sum of all voxel values in one neighbourhood (i.e. 3 × 3 × 3 box), then subtracts the central value from the above sum and divides it by 10. The absolute of the difference between the central value and the above result after dividing by 10 is restricted to a value between 0.0 and 1.0. Gradient type 3 takes the sum of all voxel values in two neighbourhoods (i.e. 5 × 5 × 5 box), then subtracts the central value from the above sum and divides it by 70. As for type 2, the absolute of this difference is also restricted to a value between 0.0 and 1.0. In gradient types 2 and 3, the 'gradient' value is shifted according to the voxel value (i.e. gradients for lower voxel values will be smaller compared to those for higher voxel values). For our case study, type 2 gradient thresholding combined with values thresholding were used to select the range of the histogram for segmenting out the right cheek complex. Gradient thresholding is also sensitive to noise and intensity inhomogeneities. There is no best combination when it comes to which gradient type one should use. Generally speaking, most boundaries can be clarified using gradient types 2 and 3. By contrast, gradient type 1 is more applicable with a cleaner dataset with fewer phases, such as medical datasets.
The segmented right cheek unit has also been extracted as surface meshes (figure 3), which also illustrates the use of the new mesh simplification function in Drishti v. 2.7. In this case, the inner surface of the cheek unit is used to test different simplification procedures, because of its more complex structure (the upper jaw cartilage of the fish has surface ossification and is fused to the inside of the cheek bones). The surface mesh data went through different stages of simplification (see electronic supplementary royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 7: 201033 material) to produce good quality surface models with relatively small file size. This allows faster physical analysing (e.g. finite-element analysis) and three-dimensional printing. Different mesh simplification gives different number of vertices, which affects the presentation of the information and the detection of details (figure 3a,c,e,g). When more triangles have been decimated, the loss of information results in a reduction in file size, which allows for easier sharing of the digital three-dimensional model [20], or uploading the model to an open data repository for educating the public [21]. By comparing the number of details lost or preserved in our case study (arrows in figure 3), a mesh smoothing factor of 2 and 50% decimation is recommended (figure 3e,f). This halves the data size but still preserves most of the information. Threedimensional models can then be used to generate three-dimensional printouts [39] or three-dimensional portable documents [40], which can test a previous hypothesis, help with functional morphology investigations, or be used for science communication and public outreach [41].

Conclusion
The new tools and proposed workflow to segment volumetric data in the latest   [31]. Arrows indicate the reference points for detailed structure compared under the different simplified procedures (see discussion in the text).
royalsocietypublishing.org/journal/rsos R. Soc. Open Sci. 7: 201033 can be used as a framework to segment computed tomography data and other forms of volumetric data and is widely applicable in biological, medical and earth science research. Our work will facilitate further crossdiscipline collaborations using three-dimensional segmentation of computed tomography data.
Data accessibility. Three-dimensional surface mesh data of the segmented right cheek complex with different levels of simplification and the cropped raw data of the region-of-interest including the right cheek complex are available from the