Skip main navigationJournal menuClose Drawer MenuOpen Drawer Menu
Skip main navigationJournal menuClose Drawer MenuOpen Drawer Menu
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
Restricted accessArticles

Stork data scheduler: mitigating the data bottleneck in e-Science

Published:28 August 2011https://doi.org/10.1098/rsta.2011.0148

    Abstract

    In this paper, we present the Stork data scheduler as a solution for mitigating the data bottleneck in e-Science and data-intensive scientific discovery. Stork focuses on planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked inputs/outputs for petascale distributed e-Science applications. Unlike existing approaches, Stork treats data resources and the tasks related to data access and movement as first-class entities just like computational resources and compute tasks, and not simply the side-effect of computation. Stork provides unique features such as aggregation of data transfer jobs considering their source and destination addresses, and an application-level throughput estimation and optimization service. We describe how these two features are implemented in Stork and their effects on end-to-end data transfer performance.

    Footnotes

    One contribution of 12 to a Theme Issue ‘e-Science: novel research, new science and enduring impact’.

    References

    • 1
      Hey T.& Trefethen A.. 2003The data deluge: an e-science perspective. Grid computing—making the global infrastructure a realityNew York, NYWiley and Sonsch. 36, pp. 809–824. Google Scholar
    • 2
      CCSP. 2003Strategic plan for the US Climate Change Science ProgramWashington, DCCCSP. A Report by the Climate Change Science Program and the Subcommittee on Global Change Research. Google Scholar
    • 3
      Kosar T., Balman M., Suslu I., Yildirim E.& Yin D.. 2009Data-aware distributed computing with stork data scheduler. In Proc. SEE-GRID-SCI’09, Istanbul, Turkey, 9–10 December 2009. Google Scholar
    • 4
      Kosar T.& Livny M.. 2004Stork: making data placement a first class citizen in the grid. In Proc. 24th Int. Conf. on Distributed Computing Systems (ICDCS’04), Tokyo, Japan, 23–26342–349Washington, DCIEEE Computer Society. Google Scholar
    • 5
      SCOOPSURA Coastal Ocean Observing and Prediction. See http://scoop.sura.org/. Google Scholar
    • 6
      UCoMSUbiquitous Computing and Monitoring System for discovery and management of energy resources. See http://www.ucoms.org/overview.html. Google Scholar
    • 7
      DPOSSThe Palomar digital sky survey. See http://www.astro.caltech.edu/~george/dposs/. Google Scholar
    • 8
      WCERWisconsin Center for Education Research digital video processing project. See http://www.wcer.wisc.edu/. Google Scholar
    • 9
      NumRelNumerical Relativity at LSU. See http://www.cct.lsu.edu/numerical/. Google Scholar
    • 10
      MSCFDMultiscale Computational Fluid Dynamics at LSU. See http://www.cct.lsu.edu/IGERT/. Google Scholar
    • 11
      DOE. 2004Report from the DOE Office of Science Data-Management Workshops, San Jose, CA, and Chicago, IL, March–May 2004Washington, DCThe Office of Science Data-Management Challenge. Google Scholar
    • 12
      Tierney B. L., Lee J., Crowley B., Holding M., Hylton J.& Drake F. L.. 1999A network-aware distributed storage cache for data-intensive environments. In Proc. 8th Int. Symp. on High Performance Distributed Computing (HPDC’99), Redondo Beach, CA, 3–6 August 1999185–189Washington, DCIEEE Computer Society. Google Scholar
    • 13
      Johnston W. E., Gannon D., Nitzberg B., Tanner L. A., Thigpen B.& Woo A.. 2000Computing and data grids for science and engineeringDallas, TXSupercomputing. Crossref, Google Scholar
    • 14
      Allcock B., et al.2001Secure, efficient data transport and replica management for high-performance data-intensive computing. In Proc. 18th Conf. IEEE Mass Storage Systems, San Diego, CA, 17–20 April 2001Washington, DCIEEE Computer Society. Google Scholar
    • 15
      Allcock B., et al.2002Data management and transfer in high performance computational grid environments. Parallel Comput. 28, 749-771. Crossref, ISI, Google Scholar
    • 16
      Allcock B., et al.2001High-performance remote access to climate simulation data: a challenge problem for data grid technologies. In Proc. 2001 ACM/IEEE Conf. on Supercomputing, Denver, CO, 10–16 November 2001Washington, DCIEEE Computer Society. Google Scholar
    • 17
      Ranganathan R.& Foster I.. 2002Decoupling computation and data scheduling in distributed data-intensive applications. In Proc. 11th IEEE Int. Symp. on High Performance Distributed Computing (HPDC-11), Edinburgh, UK, 23–26 July 2002352Washington, DCIEEE Computer Society. Google Scholar
    • 18
      Ranganathan R.& Foster I.. 2004Computation scheduling and data replication algorithms for data Grids. Grid Resource Management: State of the Art and Future Trends, Nabrzyski J., Schopf J. M.& Weglarz J.359–373Norwell, MAKluwer Academic Publishers. Google Scholar
    • 19
      Venugopal S., Buyya R.& Winton L.. 2004A grid service broker for scheduling distributed data-oriented applications on global grids. In Proc. 2nd Workshop on Middleware for Grid Computing, Toronto, Canada, 18–22 October 200475–80New York, NYACM. Google Scholar
    • 20
      ROOT. 2006An object oriented data analysis framework. See http://root.cern.ch. Google Scholar
    • 21
      Ravi M. K., Cynthia H. S.& William E. A.. 2002Reliable file transfer in Grid environments. In Proc. 27th Annual IEEE Conf. on Local Computer Networks, Tampa, FL, 6–8 November 2002737–738Washington, DCIEEE Computer Society. Google Scholar
    • 22
      Koranda S.& Moe M.. 2007Lightweight data replicator. See http://www.ligo.caltech.edu/docs/G/G030623-00/G030623-00.pdf. Google Scholar
    • 23
      Chervenak A., Schuler C., Kesselman C., Koranda S.& Moe B.. 2005Wide area data replication for scientific collaborations. In Proc. 6th IEEE/ACM Int. Workshop on Grid Computing Seattle, WA, 13–14 November 2005Washington, DCIEEE Computer Societydoi:10.1109/GRID.2005.1542717 (doi:10.1109/GRID.2005.1542717). Crossref, Google Scholar
    • 24
      Beck M., Elwasif W. R., Plank J.& Moore T.. 1999The internet backplane protocol: storage in the network. In Proc. 1999 Network Storage Symp. NetStore99, Seattle, WA, 14–15 October 1999Washington, DCIEEE Computer Society. Google Scholar
    • 25
      SRBThe Storage Resource Broker. See http://www.sdsc.edu/srb. Google Scholar
    • 26
      iRODSThe Integrated Rule Oriented Data System. See http://www.irods.org/. Google Scholar
    • 27
      SRMThe Storage Resource Managers. See http://sdm.lbl.gov/srm. Google Scholar
    • 28
      Thain D., Arpaci Dusseau A., Bent J.& Livny M.. 2004Explicit control in a batch aware distributed file system. In Proc. 1st USENIX/ACM Conf. on Networked Systems Design and Implementation, San Francisco, CA, 29–31 March 2004Berkeley, CAUSENIX. Google Scholar
    • 29
      Bent J.. 2005Data-driven batch scheduling. PhD thesis, University of Wisconsin-Madison, Madison, USA. Google Scholar
    • 30
      Stockinger H.. 2005Data management in data grids: habilitation overview. Report by the Research Laboratory for Computational Technologies and Applications, University of Vienna. Google Scholar
    • 31
      Stockinger H., Laure E.& Stockinger K.. 2005Performance engineering in data Grids. J. Concurrency Comput. Practice Exp. 17, 171-191doi:10.1002/cpe.923 (doi:10.1002/cpe.923). Crossref, ISI, Google Scholar
    • 32
      Stockinger K., Schikuta E., Stockinger H.& Willers I.. 2001Towards a cost model for distributed and replicated data stores. J. Concurrency Comput. Practice Exp.. Google Scholar
    • 33
      Crowcroft J.& Oechslin P.. 1998Differentiated end-to-end Internet services using a weighted proportional fair sharing TCP. ACM SIGCOMM Comput. Commun. Rev. 28, 53-69doi:10.1145/293927.293930 (doi:10.1145/293927.293930). Crossref, Google Scholar
    • 34
      Hacker T. J., Noble B. D.& Atley D.. 2002The end-to-end performance effects of parallel TCP sockets on a lossy wide area network. In Proc. 16th Int. Parallel and Distributed Processing Symposium, Fort Lauderdale, FL, 15–19 April 2002314Washington, DCIEEE Computer Society. Google Scholar
    • 35
      Lu D., Qiao Y., Dinda P. A.& Stamante F. E.. 2005Modeling and taming parallel TCP on the wide area network. In Proc. 19th Int. Parallel and Distributed Processing Symposium, Denver, CO, 4–8 April 2005682Washington, DCIEEE Computer Society. Google Scholar
    • 36
      Altman E., Barman D., Tuffin T.& Voinovic M.. 2006Parallel TCP sockets: simple model, throughput and validation. In Proc. INFOCOM 2006 25th IEEE Int. Conf. Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, Barcelona, Spain, 23–29 April 20061–12Washington, DCIEEE Computer Society. Google Scholar
    • 37
      Kosar T.. 2005Data placement in widely distributed systems. PhD thesis, University of Wisconsin-Madison, Madison, USA. Google Scholar
    • 38
      Kosar T., Kola G., Livny M., Brunner R. J.& Remijan M.. 2005Reliable, automatic transfer and processing of large scale astronomy data sets. In Proc. Astronomical Data Analysis Software and Systems (ADASS) XV, San Lorenzo de El Escorial, Spain, 2–5 October 2005. Google Scholar
    • 39
      Kola G., Kosar T.& Livny M.. 2004A Fully automated fault-tolerant system for distributed video processing and off-site replication. In Proc. 14th ACM Int. Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2004), Cork, Ireland, 16–18 June 2004New York, NYACM. Google Scholar
    • 40
      Ceyhan E., Allen G., White C.& Kosar T.. 2008A grid-enabled workflow system for reservoir uncertainty analysis. In Proc. 6th Int. Workshop Challenges of Large Applications in Distributed Environments (CLADE 2008), Boston, MA, 23–27 June 2008New York, NYACM. Google Scholar
    • 41
      Yildirim E., Yin D.& Kosar T.. InpressPrediction of optimal parallelism level in wide area data transfers. IEEE Trans. Parallel Distrib. Syst.. ISI, Google Scholar
    • 42
      LUSTRE Cluster File System, IncLustre: a scalable, high performance file system. See http://www.Lustre.org/. Google Scholar
    Previous Article
    Next Article