Scribd Logo
Upload

Welcome to Scribd!

  • 66 views

    Distributed Systems Group: Prof - Dr.ir. Dick H.J. Epema 5 September 2016

    Uploaded by

    m4t4ron
    The document provides information about the Distributed Systems Group including its people, teaching activities, research topics, and experimentation infrastructure. It discusses the group's work in areas like distributed algorithms, cloud computing, big data, online social networks, and cooperative systems. Specifically, it summarizes research on processor co-allocation across multiple clusters, graph processing, and the Tribler peer-to-peer system including its use of trust and reputation systems. The document encourages students to consider doing their master's thesis with the group.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Distributed Systems Group: Prof - Dr.ir. Dick H.J. Epema 5 September 2016

    Uploaded by

    m4t4ron
    66 views21 pages
    The document provides information about the Distributed Systems Group including its people, teaching activities, research topics, and experimentation infrastructure. It discusses the group's work in areas like distributed algorithms, cloud computing, big data, online social networks, and cooperative systems. Specifically, it summarizes research on processor co-allocation across multiple clusters, graph processing, and the Tribler peer-to-peer system including its use of trust and reputation systems. The document encourages students to consider doing their master's thesis with the group.

    Original Description:

    had to do this crap

    Original Title

    Pitch DS-MSc-20160905

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document provides information about the Distributed Systems Group including its people, teaching activities, research topics, and experimentation infrastructure. It discusses the group's work in areas like distributed algorithms, cloud computing, big data, online social networks, and cooperative systems. Specifically, it summarizes research on processor co-allocation across multiple clusters, graph processing, and the Tribler peer-to-peer system including its use of trust and reputation systems. The document encourages students to consider doing their master's thesis with the group.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    66 views21 pages

    Distributed Systems Group: Prof - Dr.ir. Dick H.J. Epema 5 September 2016

    Uploaded by

    m4t4ron
    The document provides information about the Distributed Systems Group including its people, teaching activities, research topics, and experimentation infrastructure. It discusses the group's work in areas like distributed algorithms, cloud computing, big data, online social networks, and cooperative systems. Specifically, it summarizes research on processor co-allocation across multiple clusters, graph processing, and the Tribler peer-to-peer system including its use of trust and reputation systems. The document encourages students to consider doing their master's thesis with the group.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    You are on page 1of 21

    Distributed Systems Group

    Prof.dr.ir. Dick H.J. Epema


    5 September 2016

    5 September 2016
    1

    The Distributed Systems Group: People

    Dick Epema

    Alexandru Iosup

    Johan Pouwelse

    Cloud Computing Online Social Networks


    Cloud Computing
    Cybercurrencies
    Big Data
    e-Science
    Applied Security
    Online Social Networks Online Gaming

    5 September 2016

    The Distributed Systems Group: Teaching


    MSc courses:

    IN4150: Distributed Algorithms (Q2)

    IN4391: Distributed Computing Systems (Q3) (core Data Sc.&Techn.)

    IN4049: High-Performance Computing (Q1)

    IN4392: Seminar Cloud Computing (Q1)

    IN4253: Applied Security Analysis (Q3)

    5 September 2016

    (core Software Techn.)

    Example Distributed Algorithms (IN4150):


    detecting global states
    Problem:

    record the state of a distributed application


    (in an asynchronous system)

    Applications: debugging, detecting stable properties


    Issues:
    1.recording the states of the processes at the same time
    2.recording the messages that are in transit

    Real-world example:
    o recording the amounts of money in bank accounts
    at 0:00 AM on Jan 1st by the tax authority

    5 September 2016

    Topics Distributed Computing (IN4391)


    Communication:
    Scheduling:
    Fault tolerance:
    Big Data:
    Use case 1:
    Use case 2:

    5 September 2016

    naming and messaging


    distributed workloads and
    resource management
    reliability, availability
    consistency, replication,
    distributed file systems
    scientific/engineering data
    processing
    massive multiplayer online gaming

    The Distributed Systems Group: type of Research


    Our research is:
    fundamental: devise new application-independent concepts
    in distributed systems
    experimental: show the value of new concepts in
    prototypes or real deployments

    5 September 2016

    Experimentation: DAS-5
    UvA/MultimediaN (62)

    VU (136 CPUs)

    UvA (36)
    System purely for CS research
    Operational since June 2015
    Specs:
    3,200 cores (8-core CPUs)

    SURFnet6

    10 Gb/s lambdas

    TU Delft (48)
    5 September 2016

    Astron (9)

    2.4 GHz CPUs


    accelerators (GPUs)
    800 TB storage
    10 Gb/s Infiniband
    Gb Ethernet

    Leiden (48)
    7

    The Distributed Systems Group:


    Research Topics
    1. Resource Management and Scheduling
    2. Big Data Processing (e.g., Graph Processing)
    3. Cooperative Systems

    5 September 2016

    The KOALA multicluster scheduler


    deployed
    on the DAS
    since 2005

    parallel MPI applications


    workflows
    MapReduce applications

    Local
    Resource
    Managers

    5 September 2016

    LRM

    information
    service

    LRM

    LRM

    LRM

    Processor co-allocation (1)


    Idea: split up a parallel application in components and
    schedule these in different clusters
    Reasons:
    o to benefit from available resources (e.g., processors, data)
    o application characteristics (e.g., simulation in one location,
    visualization in another)

    job

    Main problems:
    o need to coordinate allocations by
    autonomous resource managers
    o schedule efficiently

    5 September 2016

    clusters

    10

    Co-allocation (2): wide-area communication


    Co-allocated applications are less efficient due to the
    relatively slow wide-area communications

    execution
    time
    (seconds)

    32 CPUs

    1
    2
    3
    4
    number of clusters combined
    5 September 2016

    11

    Co-allocation (3): slowdown


    Slowdown of a job:
    execution time on multicluster
    execution time on single cluster

    (>1 usually)

    Processor co-allocation is a trade-off between


    + faster access to more capacity
    - longer execution times

    Many scheduling policies possible for co-allocation


    Main finding: co-allocation is beneficial when slowdown 1.20
    5 September 2016

    12

    Big Data Processing


    graph processing

    5 September 2016

    13

    Cooperative Systems (1): Tribler


    Is based on the BitTorrent P2P file-sharing system
    Uses an epidemic protocol for peer and content discovery
    Was first released on 17 March 2006 (1,500,000+ downloads)
    Enables video-on-demand and live streaming
    Is our research vehicle for P2P, Online Social Network
    Current focus: reputation, trust, cybercurrencies
    Download at www.tribler.org

    5 September 2016

    14

    Cooperative Systems (2): trust/reputation


    Problems:
    o why help others downloading in P2P systems?
    o why contribute to Wikipedia?
    o why be good and pay taxes?

    Solution: create a trust system without central control


    o record decentralized interaction history
    o disseminate this history in an attack-resilient way

    In Tribler:
    o maintain records of uploads and downloads
    o compute reputations from resulting graph

    5 September 2016

    15

    Cooperative Systems (3): the blockchain

    the blockchain

    block 43

    5 September 2016

    block 44
    De blockchain

    block 45

    16

    Organized by the DS Group

    icpe2016.spec.org
    5 September 2016

    17

    Doing your MSc in the DS Group


    requires you to do at least 2 of the DS MSc courses
    is hard work (experiments!!), were not easily satisfied
    will push you to a high grade (or, higher than you expected)
    lets you collaborate with other MSc and PhD students
    learns you to become an independent engineer/researcher
    can be done within or outside the university (institutes (NIKHEF)/
    companies (Solvinity))
    may lead to a conference paper, but much depends on you

    5 September 2016

    18

    Some MSc Research Topics for You

    @large

    Data Center
    Workloads

    Cloud Res.Man.
    Management

    Gaming
    @large

    Big Data: Graphs

    Cybercurrencies
    Tribler

    Big Data:
    Elastic MapRed

    5 September 2016

    19

    More information
    MSc coordinator: Johan Pouwelse
    Some previous MSc theses:
    www.ds.ewi.tudelft.nl/epema/teaching

    DS publications
    http://publications.st.ewi.tudelft.nl

    Home pages:

    www.ds.ewi.tudelft.nl

    www.ds.ewi.tudelft.nl/epema

    www.ds.ewi.tudelft.nl/~iosup

    www.ds.ewi.tudelft.nl/pouwelse

    Web sites:
    KOALA:

    www.st.ewi.tudelft.nl/koala

    DAS5:

    www.cs.vu.nl/das5

    Tribler:

    www.tribler.org

    5 September 2016

    20

    DS Tag Cloud

    5 September 2016

    21

    You might also like

    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy