Existing data movement tools are still bound by major inefficiencies when running on multicore systems. To address these inefficiencies and limitations, DOE’s Advanced Scientific Computing Research (ASCR) office has funded Fermilab and Brookhaven National Laboratory to collaboratively work on the Multicore-Aware Data Transfer Middleware (MDTM) project. MDTM aims to accelerate data movement toolkits on multicore systems. A prototype version of MDTM is currently undergoing evaluation and enhancement. [1]

Figure 1. Multicore-Aware Data Transfer Middleware (MDTM)

Multicore-Aware Data Transfer Middleware (MDTM) is shown in figure 1. MDTM aims to accelerate data movement toolkits on multicore systems. Essentially, the MDTM project consists of two major components:

• MDTM middleware services to harness multicore parallelism and make intelligent decisions that align CPU, memory, and I/O device operations in a manner which optimizes performance for higher layer applications.
• MDTM-enabled data transfer applications (client or server) that can utilize MDTM middleware to reserve and manage multiple CPU cores, memory, network devices, disk storage as an integrated resource entity, thus achieving higher throughput and enhanced quality of service when compared with existing approaches.[2]

Multicore-Aware Data Transfer Middleware (MDTM) is a type of middleware specifically designed to optimize data transfer performance in networked systems that utilize multicore processors. It aims to leverage the processing power of multiple cores to achieve faster and more efficient data transfer between different nodes in a network.

Traditional data transfer mechanisms often rely on a single core or thread to handle the entire transfer process. This approach can limit the overall performance, especially in high-speed networks or when dealing with large data sets. Multicore-Aware Data Transfer Middleware addresses this limitation by distributing the data transfer workload across multiple cores, enabling parallel processing and improving overall transfer throughput.

The key features and techniques employed by MDTM typically include:

• Parallelization
• Load balancing
• Data prefetching
• Buffer management
• Network protocol optimization

By employing these techniques, Multicore-Aware Data Transfer Middleware enhances the overall performance and efficiency of data transfers in networked systems. It leverages the parallel processing capabilities of multicore architectures to achieve faster transfer speeds, lower latency, and better resource utilization in data-intensive applications or high-speed networks.

References:

1. https://inspirehep.net/literature/1480095
2. https://indico.cern.ch/event/304944/contributions/1672434/contribution.pdf

Cite this article:

Gokula Nandhini K (2023), Multicore-Aware Data Transfer Middleware (MDTM), Anatechmaz, pp.111