Linux 4.15> does not allow mmap with a large address

[soramichi]

Maybe related to #9 but a different issue.

Since linux 4.15, mmap checks if the addresses given as parameters are not harmful in the sense that they do not intentionally set flag bits embedded in the addresses (such as NX bit-related ones).
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/arch/x86/mm/mmap.c?h=v4.15.8&id=be62a32044061cb4a3b70a10598e093f1319102e

Due to this change, mc-mapping with the -x option does not work for linux 4.15> because 0x1000000080000000 used as the offset parameter of mmap is condemned by this check and the mmap returns EINVAL.

[heechul]

Thanks for the update.

We are currently testing a new detector that uses the safer pagemap interface instead of using /dev/mem. The new detector can be found in the repository: mc-mapping-pagemap.c.

The following is the result of the new detector on the Nehalem platform we used in the original PALLOC paper, which clearly shows bit 12,13,19,20 are used for the mapping. We plan to remove the old /dev/mem version in the future.

$ sudo chrt -f 1 ./mc-mapping-pagemap -p 0.7 -n 3
mem_size (MB): 2756
allocation complete.
worker thread begins
worker thread begins
worker thread begins
Bit6: 299.67 MB/s, 213.57 ns
Bit7: 307.61 MB/s, 208.05 ns
Bit8: 295.97 MB/s, 216.24 ns
Bit9: 297.84 MB/s, 214.88 ns
Bit10: 300.66 MB/s, 212.86 ns
Bit11: 245.89 MB/s, 260.28 ns
Bit12: 792.58 MB/s, 80.75 ns
Bit13: 789.23 MB/s, 81.09 ns
Bit14: 296.21 MB/s, 216.06 ns
Bit15: 294.19 MB/s, 217.55 ns
Bit16: 240.98 MB/s, 265.58 ns
Bit17: 294.20 MB/s, 217.54 ns
Bit18: 294.07 MB/s, 217.64 ns
Bit19: 789.05 MB/s, 81.11 ns
Bit20: 789.15 MB/s, 81.10 ns
Bit21: 294.17 MB/s, 217.56 ns
Bit22: 240.98 MB/s, 265.58 ns
Bit23: 294.10 MB/s, 217.62 ns

[soramichi]

Hi, thanks for the information.

the new detector does not seem working well in my environment. I can provide further information if it would help improving the detector. The machine has only 1 DIMM and I set g_cache_num_ways to match the actual associativity. Note that the machine is different from the one I used in #9.

I guess memory accesses generated by mc-mapping-pagemap are served by the cache as their latency is reasonably shorter than the one measured by Intel Memory Latency Checker (65.1 ns).

$ cat /proc/cpuinfo | grep "model name" | uniq
model name	: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

$ cat /sys/devices/system/cpu/cpu0/cache/index3/ways_of_associativity 
20

$  sudo chrt -f 1 ./mc-mapping-pagemap -p 0.7 -n 3
mem_size (MB): 5518
allocation complete.
worker thread begins
worker thread begins
worker thread begins
Bit6: 1228.09 MB/s, 52.11 ns
Bit7: 1441.79 MB/s, 44.39 ns
Bit8: 1619.70 MB/s, 39.51 ns
Bit9: 1660.54 MB/s, 38.54 ns
Bit10: 1663.63 MB/s, 38.47 ns
Bit11: 1647.35 MB/s, 38.85 ns
Bit12: 1726.24 MB/s, 37.07 ns
Bit13: 1709.34 MB/s, 37.44 ns
Bit14: 1902.05 MB/s, 33.65 ns
Bit15: 1876.14 MB/s, 34.11 ns
Bit16: 1936.24 MB/s, 33.05 ns
Bit17: 1901.17 MB/s, 33.66 ns
Bit18: 1714.88 MB/s, 37.32 ns
Bit19: 1699.40 MB/s, 37.66 ns
Bit20: 1540.65 MB/s, 41.54 ns
Bit21: 1557.51 MB/s, 41.09 ns 
Bit22: 1924.62 MB/s, 33.25 ns
Bit23: 1703.65 MB/s, 37.57 ns

$ sudo ./mlc
Intel(R) Memory Latency Checker - v3.1a
Measuring idle latencies (in ns)...
	Memory node
Socket	     0	
     0	  65.1

[travisdowns]

For what it's worth, here are my results on an i7-6700HQ with two R2x8 DDR4 DIMMs (each with 16 banks and 4 bank groups):

$ sudo chrt -f 1 ./mc-mapping-pagemap -p 0.7 -n 3
mem_size (MB): 11136
allocation complete.
worker thread begins
worker thread begins
worker thread begins
Bit6: 4671.85 MB/s, 13.70 ns
Bit7: 4818.67 MB/s, 13.28 ns
Bit8: 4750.99 MB/s, 13.47 ns
Bit9: 4751.69 MB/s, 13.47 ns
Bit10: 4735.04 MB/s, 13.52 ns
Bit11: 4818.97 MB/s, 13.28 ns
Bit12: 4729.72 MB/s, 13.53 ns
Bit13: 4817.95 MB/s, 13.28 ns
Bit14: 4728.85 MB/s, 13.53 ns
Bit15: 4817.50 MB/s, 13.28 ns
Bit16: 1966.58 MB/s, 32.54 ns
Bit17: 4095.22 MB/s, 15.63 ns
Bit18: 3948.74 MB/s, 16.21 ns
Bit19: 3857.12 MB/s, 16.59 ns
Bit20: 3908.53 MB/s, 16.37 ns
Bit21: 1059.46 MB/s, 60.41 ns
Bit22: 918.14 MB/s, 69.71 ns
Bit23: 525.82 MB/s, 121.72 ns

I'm not sure how to interpret the results: the latencies for almost all bits seem way to low to reflect misses to main memory. Note that Skylake has a hashed L3, so you can't rely on simple ways/sets logic to force misses by overloading a way: you would need to know the hashing function (it has been reverse engineered elsewhere).