Depth image super-resolution reconstruction based on a modified joint trilateral filter

Depth image super-resolution (SR) is a technique that uses signal processing technology to enhance the resolution of a low-resolution (LR) depth image. Generally, external database or high-resolution (HR) images are needed to acquire prior information for SR reconstruction. To overcome the limitations, a depth image SR method without reference to any external images is proposed. In this paper, a high-quality edge map is first constructed using a sparse coding method, which uses a dictionary learned from the original images at different scales. Then, the high-quality edge map is used to guide the interpolation for depth images by a modified joint trilateral filter. During the interpolation, some information of gradient and structural similarity (SSIM) are added to preserve the detailed information and suppress the noise. The proposed method can not only preserve the sharpness of image edge, but also avoid the dependence on database. Experimental results show that the proposed method is superior to some state-of-the-art depth image SR methods.


Recommendation?
Accept with minor revision (please list in comments)

Comments to the Author(s)
The work at hand proposes an interesting method for deep image SR which needs neither the external example database nor the assistance of the registered HR color image. The core of the proposed method was named Edge Guided Joint Trilateral Filter. The whole paper is clear, well written, and easy to follow the motivation. The theory is correctly derived and the experimental analysis is sufficient. Overall I am not convinced of this manuscript in its current form. 1. The number of the references needs to be reorder. In Fig.3, the "Timofe [11]" is should be [12]…… 2. Overall the quality of the Figures is not satisfying. For example, the Fig.2 should be included as vector graphics or in an adequate resolution, such that details are not covered by compression artifacts. Or, maybe, the low resolution pictures which look not clear are waiting for SR? 3. Is there any discussion about table 6, why the running time of the proposed method is higher than most other methods?
Decision letter (RSOS-181074.R0) 24-Oct-2018 Dear Professor Zhou, The editors assigned to your paper ("Deep Image Super-Resolution Reconstruction Based on an Improved Joint Trilateral Filter") have now received comments from reviewers. We would like you to revise your paper in accordance with the referee and Associate Editor suggestions which can be found below (not including confidential reports to the Editor). Please note this decision does not guarantee eventual acceptance.
Please submit a copy of your revised paper before 16-Nov-2018. Please note that the revision deadline will expire at 00.00am on this date. If we do not hear from you within this time then it will be assumed that the paper has been withdrawn. In exceptional circumstances, extensions may be possible if agreed with the Editorial Office in advance. We do not allow multiple rounds of revision so we urge you to make every effort to fully address all of the comments at this stage. If deemed necessary by the Editors, your manuscript will be sent back to one or more of the original reviewers for assessment. If the original reviewers are not available, we may invite new reviewers.
To revise your manuscript, log into http://mc.manuscriptcentral.com/rsos and enter your Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions." Under "Actions," click on "Create a Revision." Your manuscript number has been appended to denote a revision. Revise your manuscript and upload a new version through your Author Centre.
When submitting your revised manuscript, you must respond to the comments made by the referees and upload a file "Response to Referees" in "Section 6 -File Upload". Please use this to document how you have responded to the comments, and the adjustments you have made. In order to expedite the processing of the revised manuscript, please be as specific as possible in your response.
In addition to addressing all of the reviewers' and editor's comments please also ensure that your revised manuscript contains the following sections as appropriate before the reference list: • Ethics statement (if applicable) If your study uses humans or animals please include details of the ethical approval received, including the name of the committee that granted approval. For human studies please also detail whether informed consent was obtained. For field studies on animals please include details of all permissions, licences and/or approvals granted to carry out the fieldwork.
• Data accessibility It is a condition of publication that all supporting data are made available either as supplementary information or preferably in a suitable permanent repository. The data accessibility section should state where the article's supporting data can be accessed. This section should also include details, where possible of where to access other relevant research materials such as statistical tools, protocols, software etc can be accessed. If the data have been deposited in an external repository this section should list the database, accession number and link to the DOI for all data from the article that have been made publicly available. Data sets that have been deposited in an external repository and have a DOI should also be appropriately cited in the manuscript and included in the reference list.
If you wish to submit your supporting data or code to Dryad (http://datadryad.org/), or modify your current submission to dryad, please use the following link: http://datadryad.org/submit?journalID=RSOS&manu=RSOS-181074 • Competing interests Please declare any financial or non-financial competing interests, or state that you have no competing interests.
• Authors' contributions All submissions, other than those with a single author, must include an Authors' Contributions section which individually lists the specific contribution of each author. The list of Authors should meet all of the following criteria; 1) substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published.
All contributors who do not meet all of these criteria should be included in the acknowledgements.
We suggest the following format: AB carried out the molecular lab work, participated in data analysis, carried out sequence alignments, participated in the design of the study and drafted the manuscript; CD carried out the statistical analyses; EF collected field data; GH conceived of the study, designed the study, coordinated the study and helped draft the manuscript. All authors gave final approval for publication.
• Acknowledgements Please acknowledge anyone who contributed to the study but did not meet the authorship criteria.
• Funding statement Please list the source of funding for each author.
Please note that Royal Society Open Science charge article processing charges for all new submissions that are accepted for publication. Charges will also apply to papers transferred to Royal Society Open Science from other Royal Society Publishing journals, as well as papers submitted as part of our collaboration with the Royal Society of Chemistry (http://rsos.royalsocietypublishing.org/chemistry). If your manuscript is newly submitted and subsequently accepted for publication, you will be asked to pay the article processing charge, unless you request a waiver and this is approved by Royal Society Publishing. You can find out more about the charges at http://rsos.royalsocietypublishing.org/page/charges. Should you have any queries, please contact openscience@royalsociety.org.
Once again, thank you for submitting your manuscript to Royal Society Open Science and I look forward to receiving your revision. If you have any questions at all, please do not hesitate to get in touch. Comments to the Author(s) At present, with the popularity of deep camera systems, such as Microsoft's kinect, Apple's iphoneX and other consumer devices have already adopted a deep camera system, but the common problem is low precision. Based on the above background, this paper has carried out research on the super-resolution enhancement method of depth image, which has good frontier and practicality. This paper proposes a depth image enhancement method based on an improved joint trilateral filter. More interesting is that this manuscript proposes a method for superresolution image reconstruction without relying on external databases or high-resolution images to obtain prior information. The overall logical framework of the paper is clear and the overall writing is better. The method description, demonstration and experiments are complete, and the results are credible. But, some of the following questions should be considered: 1. I recommended having a native English speaker to go through this paper. 2. There are many spelling errors in the paper. For example, the characters Dl on the 5th line and the 8th line of page 4 are garbled. Please modify 3. There is an error in the chapter number of the paper. For example, there exists inconsistency in the subsection numbers in Sections 4 and 5. 4. Table 1 lists the 10 parameter values set before the experiment. How these parameter values are determined should be explained in detail. 5. There are some bugs in the running of the attached program code. It is recommended to attach a detailed program running instructions.
Generally, the manuscript is good with nice motivations, method and extensive experiments. I recommend accepting this paper after minor revision.

Comments to the Author(s)
No. The authors have revised the manuscript based on the comments. The quality of pictures should be further improved.

Recommendation?
Accept as is

Comments to the Author(s)
Your reseach job is valuable and intersting.

17-Dec-2018
Dear Professor Zhou, I am pleased to inform you that your manuscript entitled "Depth Image Super-Resolution Reconstruction Based on a Modified Joint Trilateral Filter" is now accepted for publication in Royal Society Open Science.
You can expect to receive a proof of your article in the near future. Please contact the editorial office (openscience_proofs@royalsociety.org and openscience@royalsociety.org) to let us know if you are likely to be away from e-mail contact. Due to rapid publication and an extremely tight schedule, if comments are not received, your paper may experience a delay in publication.
Royal Society Open Science operates under a continuous publication model (http://bit.ly/cpFAQ). Your article will be published straight into the next open issue and this will be the final version of the paper. As such, it can be cited immediately by other researchers. As the issue version of your paper will be the only version to be published I would advise you to check your proofs thoroughly as changes cannot be made once the paper is published.

Explanation of this revision
Thank you very much for your valuable comments, which has been very useful for improving the quality of this paper. The details of the revision are as follows: Question: 1 I recommended having a native English speaker to go through this paper.

Answer:
Thank you for your careful reading very much. The whole manuscript has been reedited by a native English speaker who fixed many grammar and expression errors.

Question:2
There are many spelling errors in the paper. For example, the characters l  D on the 5th line and the 8th line of page 4 are garbled. Please modify Answer: The characters l  D have been modified on the 4th line and the 6th line of page 3, and further, we also examined the entire manuscript and eliminated the similar problem.

Question:3
There is an error in the chapter number of the paper. For example, there exists inconsistency in the subsection numbers in Sections 4 and 5.

Answer:
Thank you for your reminder. We have reedited the subsection numbers in Sections 4 and 5. Table 1 lists the 10 parameter values set before the experiment. How these parameter values are determined should be explained in detail.

Answer:
The selection of parameters is introduced on the lines 26-30 of page 5 in detail.

Question:5
There are some bugs in the running of the attached program code. It is recommended to attach a detailed program running instructions.

Answer:
The new version of the codes has been uploaded to the Dryad website, which added the program description, and some bugs have been fixed. You can download the new codes from the URL: https://datadryad.org/review?doi=doi:10.5061/dryad.5ph7sm6