I'm trying to figure out the optimal capacity and load factor for a specific case. I think I got the gist of it, but I'd still be thankful for a confirmation from someone more knowledgable than me. :)
我想求出一个特定情况下的最优容量和负载因数。我想我明白了,但我还是要感谢那些比我更有学问的人。:)
If I know that my HashMap will fill up to contain, say, 100 objects, and will spend most of the time having 100 objects, I'm guessing that the optimal values are initial capacity 100 and load factor 1? Or do I need capacity 101, or are there any other gotchas?
如果我知道我的HashMap会填充,比如说,100个对象,并且会花费大部分时间有100个对象,我猜最优值是初始容量100和负载因子1?或者我需要容量101,或者有其他问题吗?
EDIT: OK, I set aside a few hours and did some testing. Here are the results:
编辑:好的,我留出几个小时,做了一些测试。这里是结果:
- Curiously, capacity, capacity+1, capacity+2, capacity-1 and even capacity-10 all yield exactly the same results. I would expect at least capacity-1 and capacity-10 to give worse results.
- 奇怪的是,容量、容量+1、容量+2、容量-1甚至容量-10都会产生完全相同的结果。我预计至少能力-1和能力-10会产生更坏的结果。
- Using initial capacity (as opposed to using default value of 16) gives noticable put() improvement - up to 30% faster.
- 使用初始容量(而不是使用默认值16)可以获得显著的put()改进——速度快30%。
- Using load factor of 1 gives equal performance for small number of objects, and better performance for larger number of objects (>100000). However, this does not improve proportionally to the number of objects; I suspect there is additional factor that affects the results.
- 使用负载因子1对少量对象具有相同的性能,对大量对象具有更好的性能(>100000)。然而,这并不能与对象的数量成比例地提高;我怀疑还有其他因素会影响结果。
- get() performance is a bit different for different number of objects/capacity, but though it might slightly vary from case to case, generally it's not affected by initial capacity or load factor.
- get()性能对于不同的对象/容量有一点不同,但是尽管它可能稍有不同,但通常不会受初始容量或负载因素的影响。
EDIT2: Adding some charts on my part as well. Here's the one illustrating difference between load factor 0.75 and 1, in cases where I initialize HashMap and fill it up to full capacity. On y scale is time in ms (lower is better), and x scale is size (number of objects). Since size changes linearly, the time required grows linearly as well.
在我这边也加一些图表。这里有一个说明负载因子0.75和1之间的差异的例子,在我初始化HashMap并将其填满的情况下。y标度表示ms中的时间(越低越好),x标度表示大小(对象数量)。由于尺寸线性变化,所需的时间也线性增长。
So, let's see what I got. The following two charts show the difference in load factors. First chart shows what happens when HashMap is filled to capacity; load factor 0.75 performs worse because of resizing. However, it's not consistently worse, and there are all sorts of bumps and hops - I guess that GC has a major play in this. Load factor 1.25 performs the same as 1, so it's not included in the chart.
让我们看看我得到了什么。下面两个图表显示了负载因素的差异。第一个图表显示了当HashMap被填充到容量时发生了什么;负载系数0.75由于调整大小而变得更糟。然而,它并不总是更糟,而且有各种各样的颠簸和跳跃——我猜GC在这方面起着重要作用。负载因数1.25与1执行相同,因此不在图表中。
This chart proves that 0.75 was worse due to resizing; if we fill the HashMap to half capacity, 0.75 is not worse, just... different (and it should use less memory and have unnoticably better iteration performance).
这张图证明了0.75由于尺寸调整而变差;如果我们将HashMap填满一半容量,那么0.75不是更糟,只是……不同(并且它应该使用更少的内存,并且具有更好的迭代性能)。
One more thing I want to show. This is get performance for all three load factors and different HashMap sizes. Consistently constant with a little variation, except for one spike for load factor 1. I'd really want to know what that is (probably GC, but who knows).
还有一件事我想展示。这是获取所有三个负载因素和不同HashMap大小的性能。始终如一的恒定不变,除了一个负载因子1的峰值。我很想知道那是什么(可能是GC,但谁知道呢)。
And here's the code for those interested:
以下是那些感兴趣的人的代码:
import java.util.HashMap;
import java.util.Map;
public class HashMapTest {
// capacity - numbers high as 10000000 require -mx1536m -ms1536m JVM parameters
public static final int CAPACITY = 10000000;
public static final int ITERATIONS = 10000;
// set to false to print put performance, or to true to print get performance
boolean doIterations = false;
private Map<Integer, String> cache;
public void fillCache(int capacity) {
long t = System.currentTimeMillis();
for (int i = 0; i <= capacity; i++)
cache.put(i, "Value number " + i);
if (!doIterations) {
System.out.print(System.currentTimeMillis() - t);
System.out.print("\t");
}
}
public void iterate(int capacity) {
long t = System.currentTimeMillis();
for (int i = 0; i <= ITERATIONS; i++) {
long x = Math.round(Math.random() * capacity);
String result = cache.get((int) x);
}
if (doIterations) {
System.out.print(System.currentTimeMillis() - t);
System.out.print("\t");
}
}
public void test(float loadFactor, int divider) {
for (int i = 10000; i <= CAPACITY; i+= 10000) {
cache = new HashMap<Integer, String>(i, loadFactor);
fillCache(i / divider);
if (doIterations)
iterate(i / divider);
}
System.out.println();
}
public static void main(String[] args) {
HashMapTest test = new HashMapTest();
// fill to capacity
test.test(0.75f, 1);
test.test(1, 1);
test.test(1.25f, 1);
// fill to half capacity
test.test(0.75f, 2);
test.test(1, 2);
test.test(1.25f, 2);
}
}
4 个解决方案
#1
66
Alright, to put this thing to rest, I've created a test app to run a couple of scenarios and get some visualizations of the results. Here's how the tests are done:
好了,为了让这个东西停下来,我创建了一个测试应用程序来运行一些场景并得到结果的可视化效果。下面是测试的方法:
- A number of different collection sizes have been tried: one hundred, one thousand and one hundred thousand entries.
- 已经尝试了许多不同的收集尺寸:100、1000和10万条目。
- The keys used are instances of a class that are uniquely identified by an ID. Each test uses unique keys, with incrementing integers as IDs. The
equals
method only uses the ID, so no key mapping overwrites another one. - 使用的键是由ID唯一标识的类的实例。每个测试使用唯一的键,递增的整数作为ID。equals方法只使用ID,因此没有键映射重写另一个ID。
- The keys get a hash code that consists of the module remainder of their ID against some preset number. We'll call that number the hash limit. This allowed me to control the number of hash collisions that would be expected. For example, if our collection size is 100, we'll have keys with IDs ranging from 0 to 99. If the hash limit is 100, every key will have a unique hash code. If the hash limit is 50, key 0 will have the same hash code as key 50, 1 will have the same hash code as 51 etc. In other words, the expected number of hash collisions per key is the collection size divided by the hash limit.
- 键获取一个散列代码,该散列代码由它们的ID的模块余数与某个预设值组成。我们称之为哈希极限。这使我能够控制预期的散列冲突的数量。例如,如果我们的集合大小是100,那么我们将有id从0到99的键。如果哈希限制是100,每个键都有一个唯一的哈希代码。如果哈希限制是50,那么key 0将具有与key 50相同的哈希代码,1将具有与51相同的哈希代码,等等。换句话说,每个键的哈希冲突的期望值是集合大小除以哈希限制。
- For each combination of collection size and hash limit, I've run the test using hash maps initialized with different settings. These settings are the load factor, and an initial capacity that is expressed as a factor of the collection setting. For example, a test with a collection size of 100 and an initial capacity factor of 1.25 will initialize a hash map with an initial capacity of 125.
- 对于集合大小和散列限制的每种组合,我都使用不同设置初始化的散列映射来运行测试。这些设置是负载因数,以及表示为集合设置因数的初始容量。例如,一个集合大小为100,初始容量因子为1.25的测试将初始化一个初始容量为125的散列映射。
- The value for each key is simply a new
Object
. - 每个键的值只是一个新对象。
- Each test result is encapsulated in an instance of a Result class. At the end of all tests, the results are ordered from worst overall performance to best.
- 每个测试结果都封装在一个result类的实例中。在所有测试的最后,结果都是从最糟糕的整体性能到最好的。
- The average time for puts and gets is calculated per 10 puts/gets.
- 投入和获得的平均时间是每10次投入/获得。
- All test combinations are run once to eliminate JIT compilation influence. After that, the tests are run for actual results.
- 所有的测试组合都运行一次,以消除JIT编译的影响。之后,将运行测试以获得实际结果。
Here's the class:
类:
package hashmaptest;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
public class HashMapTest {
private static final List<Result> results = new ArrayList<Result>();
public static void main(String[] args) throws IOException {
//First entry of each array is the sample collection size, subsequent entries
//are the hash limits
final int[][] sampleSizesAndHashLimits = new int[][] {
{100, 50, 90, 100},
{1000, 500, 900, 990, 1000},
{100000, 10000, 90000, 99000, 100000}
};
final double[] initialCapacityFactors = new double[] {0.5, 0.75, 1.0, 1.25, 1.5, 2.0};
final float[] loadFactors = new float[] {0.5f, 0.75f, 1.0f, 1.25f};
//Doing a warmup run to eliminate JIT influence
for(int[] sizeAndLimits : sampleSizesAndHashLimits) {
int size = sizeAndLimits[0];
for(int i = 1; i < sizeAndLimits.length; ++i) {
int limit = sizeAndLimits[i];
for(double initCapacityFactor : initialCapacityFactors) {
for(float loadFactor : loadFactors) {
runTest(limit, size, initCapacityFactor, loadFactor);
}
}
}
}
results.clear();
//Now for the real thing...
for(int[] sizeAndLimits : sampleSizesAndHashLimits) {
int size = sizeAndLimits[0];
for(int i = 1; i < sizeAndLimits.length; ++i) {
int limit = sizeAndLimits[i];
for(double initCapacityFactor : initialCapacityFactors) {
for(float loadFactor : loadFactors) {
runTest(limit, size, initCapacityFactor, loadFactor);
}
}
}
}
Collections.sort(results);
for(final Result result : results) {
result.printSummary();
}
// ResultVisualizer.visualizeResults(results);
}
private static void runTest(final int hashLimit, final int sampleSize,
final double initCapacityFactor, final float loadFactor) {
final int initialCapacity = (int)(sampleSize * initCapacityFactor);
System.out.println("Running test for a sample collection of size " + sampleSize
+ ", an initial capacity of " + initialCapacity + ", a load factor of "
+ loadFactor + " and keys with a hash code limited to " + hashLimit);
System.out.println("====================");
double hashOverload = (((double)sampleSize/hashLimit) - 1.0) * 100.0;
System.out.println("Hash code overload: " + hashOverload + "%");
//Generating our sample key collection.
final List<Key> keys = generateSamples(hashLimit, sampleSize);
//Generating our value collection
final List<Object> values = generateValues(sampleSize);
final HashMap<Key, Object> map = new HashMap<Key, Object>(initialCapacity, loadFactor);
final long startPut = System.nanoTime();
for(int i = 0; i < sampleSize; ++i) {
map.put(keys.get(i), values.get(i));
}
final long endPut = System.nanoTime();
final long putTime = endPut - startPut;
final long averagePutTime = putTime/(sampleSize/10);
System.out.println("Time to map all keys to their values: " + putTime + " ns");
System.out.println("Average put time per 10 entries: " + averagePutTime + " ns");
final long startGet = System.nanoTime();
for(int i = 0; i < sampleSize; ++i) {
map.get(keys.get(i));
}
final long endGet = System.nanoTime();
final long getTime = endGet - startGet;
final long averageGetTime = getTime/(sampleSize/10);
System.out.println("Time to get the value for every key: " + getTime + " ns");
System.out.println("Average get time per 10 entries: " + averageGetTime + " ns");
System.out.println("");
final Result result =
new Result(sampleSize, initialCapacity, loadFactor, hashOverload, averagePutTime, averageGetTime, hashLimit);
results.add(result);
//Haha, what kind of noob explicitly calls for garbage collection?
System.gc();
try {
Thread.sleep(200);
} catch(final InterruptedException e) {}
}
private static List<Key> generateSamples(final int hashLimit, final int sampleSize) {
final ArrayList<Key> result = new ArrayList<Key>(sampleSize);
for(int i = 0; i < sampleSize; ++i) {
result.add(new Key(i, hashLimit));
}
return result;
}
private static List<Object> generateValues(final int sampleSize) {
final ArrayList<Object> result = new ArrayList<Object>(sampleSize);
for(int i = 0; i < sampleSize; ++i) {
result.add(new Object());
}
return result;
}
private static class Key {
private final int hashCode;
private final int id;
Key(final int id, final int hashLimit) {
//Equals implies same hashCode if limit is the same
//Same hashCode doesn't necessarily implies equals
this.id = id;
this.hashCode = id % hashLimit;
}
@Override
public int hashCode() {
return hashCode;
}
@Override
public boolean equals(final Object o) {
return ((Key)o).id == this.id;
}
}
static class Result implements Comparable<Result> {
final int sampleSize;
final int initialCapacity;
final float loadFactor;
final double hashOverloadPercentage;
final long averagePutTime;
final long averageGetTime;
final int hashLimit;
Result(final int sampleSize, final int initialCapacity, final float loadFactor,
final double hashOverloadPercentage, final long averagePutTime,
final long averageGetTime, final int hashLimit) {
this.sampleSize = sampleSize;
this.initialCapacity = initialCapacity;
this.loadFactor = loadFactor;
this.hashOverloadPercentage = hashOverloadPercentage;
this.averagePutTime = averagePutTime;
this.averageGetTime = averageGetTime;
this.hashLimit = hashLimit;
}
@Override
public int compareTo(final Result o) {
final long putDiff = o.averagePutTime - this.averagePutTime;
final long getDiff = o.averageGetTime - this.averageGetTime;
return (int)(putDiff + getDiff);
}
void printSummary() {
System.out.println("" + averagePutTime + " ns per 10 puts, "
+ averageGetTime + " ns per 10 gets, for a load factor of "
+ loadFactor + ", initial capacity of " + initialCapacity
+ " for " + sampleSize + " mappings and " + hashOverloadPercentage
+ "% hash code overload.");
}
}
}
Running this might take a while. The results are printed out on standard out. You might notice I've commented out a line. That line calls a visualizer that outputs visual representations of the results to png files. The class for this is given below. If you wish to run it, uncomment the appropriate line in the code above. Be warned: the visualizer class assumes you're running on Windows and will create folders and files in C:\temp. When running on another platform, adjust this.
运行这个可能需要一段时间。结果印在标准输出上。你可能注意到我注释了一行。这一行调用了一个visualizer,它将结果的可视化表示输出到png文件中。这方面的类如下所示。如果您希望运行它,请取消上面代码中适当的行注释。注意:visualizer类假定您在Windows上运行,并将在C:\temp中创建文件夹和文件。在另一个平台上运行时,请对此进行调整。
package hashmaptest;
import hashmaptest.HashMapTest.Result;
import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.text.DecimalFormat;
import java.text.NumberFormat;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import javax.imageio.ImageIO;
public class ResultVisualizer {
private static final Map<Integer, Map<Integer, Set<Result>>> sampleSizeToHashLimit =
new HashMap<Integer, Map<Integer, Set<Result>>>();
private static final DecimalFormat df = new DecimalFormat("0.00");
static void visualizeResults(final List<Result> results) throws IOException {
final File tempFolder = new File("C:\\temp");
final File baseFolder = makeFolder(tempFolder, "hashmap_tests");
long bestPutTime = -1L;
long worstPutTime = 0L;
long bestGetTime = -1L;
long worstGetTime = 0L;
for(final Result result : results) {
final Integer sampleSize = result.sampleSize;
final Integer hashLimit = result.hashLimit;
final long putTime = result.averagePutTime;
final long getTime = result.averageGetTime;
if(bestPutTime == -1L || putTime < bestPutTime)
bestPutTime = putTime;
if(bestGetTime <= -1.0f || getTime < bestGetTime)
bestGetTime = getTime;
if(putTime > worstPutTime)
worstPutTime = putTime;
if(getTime > worstGetTime)
worstGetTime = getTime;
Map<Integer, Set<Result>> hashLimitToResults =
sampleSizeToHashLimit.get(sampleSize);
if(hashLimitToResults == null) {
hashLimitToResults = new HashMap<Integer, Set<Result>>();
sampleSizeToHashLimit.put(sampleSize, hashLimitToResults);
}
Set<Result> resultSet = hashLimitToResults.get(hashLimit);
if(resultSet == null) {
resultSet = new HashSet<Result>();
hashLimitToResults.put(hashLimit, resultSet);
}
resultSet.add(result);
}
System.out.println("Best average put time: " + bestPutTime + " ns");
System.out.println("Best average get time: " + bestGetTime + " ns");
System.out.println("Worst average put time: " + worstPutTime + " ns");
System.out.println("Worst average get time: " + worstGetTime + " ns");
for(final Integer sampleSize : sampleSizeToHashLimit.keySet()) {
final File sizeFolder = makeFolder(baseFolder, "sample_size_" + sampleSize);
final Map<Integer, Set<Result>> hashLimitToResults =
sampleSizeToHashLimit.get(sampleSize);
for(final Integer hashLimit : hashLimitToResults.keySet()) {
final File limitFolder = makeFolder(sizeFolder, "hash_limit_" + hashLimit);
final Set<Result> resultSet = hashLimitToResults.get(hashLimit);
final Set<Float> loadFactorSet = new HashSet<Float>();
final Set<Integer> initialCapacitySet = new HashSet<Integer>();
for(final Result result : resultSet) {
loadFactorSet.add(result.loadFactor);
initialCapacitySet.add(result.initialCapacity);
}
final List<Float> loadFactors = new ArrayList<Float>(loadFactorSet);
final List<Integer> initialCapacities = new ArrayList<Integer>(initialCapacitySet);
Collections.sort(loadFactors);
Collections.sort(initialCapacities);
final BufferedImage putImage =
renderMap(resultSet, loadFactors, initialCapacities, worstPutTime, bestPutTime, false);
final BufferedImage getImage =
renderMap(resultSet, loadFactors, initialCapacities, worstGetTime, bestGetTime, true);
final String putFileName = "size_" + sampleSize + "_hlimit_" + hashLimit + "_puts.png";
final String getFileName = "size_" + sampleSize + "_hlimit_" + hashLimit + "_gets.png";
writeImage(putImage, limitFolder, putFileName);
writeImage(getImage, limitFolder, getFileName);
}
}
}
private static File makeFolder(final File parent, final String folder) throws IOException {
final File child = new File(parent, folder);
if(!child.exists())
child.mkdir();
return child;
}
private static BufferedImage renderMap(final Set<Result> results, final List<Float> loadFactors,
final List<Integer> initialCapacities, final float worst, final float best,
final boolean get) {
//[x][y] => x is mapped to initial capacity, y is mapped to load factor
final Color[][] map = new Color[initialCapacities.size()][loadFactors.size()];
for(final Result result : results) {
final int x = initialCapacities.indexOf(result.initialCapacity);
final int y = loadFactors.indexOf(result.loadFactor);
final float time = get ? result.averageGetTime : result.averagePutTime;
final float score = (time - best)/(worst - best);
final Color c = new Color(score, 1.0f - score, 0.0f);
map[x][y] = c;
}
final int imageWidth = initialCapacities.size() * 40 + 50;
final int imageHeight = loadFactors.size() * 40 + 50;
final BufferedImage image =
new BufferedImage(imageWidth, imageHeight, BufferedImage.TYPE_3BYTE_BGR);
final Graphics2D g = image.createGraphics();
g.setColor(Color.WHITE);
g.fillRect(0, 0, imageWidth, imageHeight);
for(int x = 0; x < map.length; ++x) {
for(int y = 0; y < map[x].length; ++y) {
g.setColor(map[x][y]);
g.fillRect(50 + x*40, imageHeight - 50 - (y+1)*40, 40, 40);
g.setColor(Color.BLACK);
g.drawLine(25, imageHeight - 50 - (y+1)*40, 50, imageHeight - 50 - (y+1)*40);
final Float loadFactor = loadFactors.get(y);
g.drawString(df.format(loadFactor), 10, imageHeight - 65 - (y)*40);
}
g.setColor(Color.BLACK);
g.drawLine(50 + (x+1)*40, imageHeight - 50, 50 + (x+1)*40, imageHeight - 15);
final int initialCapacity = initialCapacities.get(x);
g.drawString(((initialCapacity%1000 == 0) ? "" + (initialCapacity/1000) + "K" : "" + initialCapacity), 15 + (x+1)*40, imageHeight - 25);
}
g.drawLine(25, imageHeight - 50, imageWidth, imageHeight - 50);
g.drawLine(50, 0, 50, imageHeight - 25);
g.dispose();
return image;
}
private static void writeImage(final BufferedImage image, final File folder,
final String filename) throws IOException {
final File imageFile = new File(folder, filename);
ImageIO.write(image, "png", imageFile);
}
}
The visualized output is as follows:
可视化输出如下:
- Tests are divided first by collection size, then by hash limit.
- 测试首先要除以集合大小,然后再除以散列限制。
- For each test, there's an output image regarding the average put time (per 10 puts) and the average get time (per 10 gets). The images are two-dimensional "heat maps" that show a color per combination of initial capacity and load factor.
- 对于每个测试,都有一个输出映像,关于平均put时间(每10次put)和平均get时间(每10次get)。这些图像是二维的“热图”,显示了初始容量和负载系数的组合颜色。
- The colours in the images are based on the average time on a normalized scale from best to worst result, ranging from saturated green to saturated red. In other words, the best time will be fully green, while the worst time will be fully red. Two different time measurements should never have the same colour.
- 图像中的颜色是基于标准化尺度上的平均时间,从最佳到最差的结果,从饱和的绿色到饱和的红色不等。换句话说,最好的时间是完全绿色的,最坏的时间是完全红色的。两个不同的时间测量不应该有相同的颜色。
- The colour maps are calculated separately for puts and gets, but encompass all tests for their respective categories.
- 颜色地图分别为put和get计算,但包含所有针对它们各自类别的测试。
- The visualizations show the initial capacity on their x axis, and the load factor on the y axis.
- 可视化显示了它们x轴上的初始容量,以及y轴上的负载因子。
Without further ado, let's take a look at the results. I'll start with the results for puts.
不用多说了,我们来看一下结果。我将从put的结果开始。
Put results
把结果
Collection size: 100. Hash limit: 50. This means each hash code should occur twice and every other key collides in the hash map.
集合大小:100。散列限制:50。这意味着每个哈希代码应该出现两次,而哈希映射中的每个其他键都发生冲突。
Well, that doesn't start off very good. We see that there's a big hotspot for an initial capacity 25% above the collection size, with a load factor of 1. The lower left corner doesn't perform too well.
这并不是很好开始。我们看到有一个较大的热点,初始容量比集合大小高出25%,负载系数为1。左下角的表现不太好。
Collection size: 100. Hash limit: 90. One in ten keys has a duplicate hash code.
集合大小:100。散列限制:90。每十个键中就有一个有重复的哈希码。
This is a slightly more realistic scenario, not having a perfect hash function but still 10% overload. The hotspot is gone, but the combination of a low initial capacity with a low load factor obviously doesn't work.
这是一个稍微真实的场景,没有完美的哈希函数,但仍然有10%的超负荷。热点消失了,但低初始容量和低负荷系数的组合显然不起作用。
Collection size: 100. Hash limit: 100. Each key as its own unique hash code. No collisions expected if there are enough buckets.
集合大小:100。散列限制:100。每个键作为它自己的唯一哈希代码。如果有足够的桶,预计不会发生碰撞。
An initial capacity of 100 with a load factor of 1 seems fine. Surprisingly, a higher initial capacity with a lower load factor isn't necessarily good.
初始容量为100的负载系数为1似乎很好。令人惊讶的是,高初始容量和低负荷系数并不一定是好的。
Collection size: 1000. Hash limit: 500. It's getting more serious here, with 1000 entries. Just like in the first test, there's a hash overload of 2 to 1.
集合大小:1000。散列限制:500。这里的问题越来越严重,有1000个条目。就像第一个测试一样,哈希重载是2比1。
The lower left corner is still not doing well. But there seems to be a symmetry between the combo of lower initial count/high load factor and higher initial count/low load factor.
左下角仍然表现不佳。但较低的初始计数/高负荷系数和较高的初始计数/低负荷系数之间似乎存在对称性。
Collection size: 1000. Hash limit: 900. This means one in ten hash codes will occur twice. Reasonable scenario regarding collisions.
集合大小:1000。散列限制:900。这意味着每十个哈希码中就有一个会出现两次。合理的场景中关于碰撞。
There's something very funny going on with the unlikely combo of an initial capacity that's too low with a load factor above 1, which is rather counter-intuitive. Otherwise, still quite symmetrical.
有一件非常有趣的事情发生了,那就是初始容量的组合不太可能是太低,负载系数大于1,这是非常反直觉的。否则,仍然很对称。
Collection size: 1000. Hash limit: 990. Some collisions, but only a few. Quite realistic in this respect.
集合大小:1000。散列限制:990。有些碰撞,但只有几个。在这方面相当现实。
We've got a nice symmetry here. Lower left corner is still sub-optimal, but the combos 1000 init capacity/1.0 load factor versus 1250 init capacity/0.75 load factor are at the same level.
这里有一个很好的对称性。左下角仍然是次优的,但是combos 1000 init容量/1.0负载因数与1250 init容量/0.75负载因数是相同的。
Collection size: 1000. Hash limit: 1000. No duplicate hash codes, but now with a sample size of 1000.
集合大小:1000。散列限制:1000。没有重复的哈希代码,但是现在示例大小为1000。
Not much to be said here. The combination of a higher initial capacity with a load factor of 0.75 seems to slightly outperform the combination of 1000 initial capacity with a load factor of 1.
这里没什么可说的。更高的初始容量和0.75的负载系数的组合似乎稍微优于1000的初始容量和1的负载系数的组合。
Collection size: 100_000. Hash limit: 10_000. Alright, it's getting serious now, with a sample size of one hundred thousand and 100 hash code duplicates per key.
_000集合大小:100。散列限制:10 _000。好了,现在它变得很重要了,每个键的样本大小是10万100个哈希代码重复。
Yikes! I think we found our lower spectrum. An init capacity of exactly the collection size with a load factor of 1 is doing really well here, but other than that it's all over the shop.
呵!我们找到了较低的频谱。加载系数为1的初始化容量在这里做得很好,但除此之外,它在整个商店都有。
Collection size: 100_000. Hash limit: 90_000. A bit more realistic than the previous test, here we've got a 10% overload in hash codes.
_000集合大小:100。散列限制:90 _000。比之前的测试更实际一点,这里我们有10%的哈希代码过载。
The lower left corner is still undesirable. Higher initial capacities work best.
左下角仍然是不可取的。较高的初始能力最好。
Collection size: 100_000. Hash limit: 99_000. Good scenario, this. A large collection with a 1% hash code overload.
_000集合大小:100。散列限制:99 _000。良好的情况下,这个问题。包含1%哈希代码过载的大型集合。
Using the exact collection size as init capacity with a load factor of 1 wins out here! Slightly larger init capacities work quite well, though.
使用准确的集合大小作为initwithload因数为1的init容量在这里胜出!不过,稍微大一点的init容量可以很好地工作。
Collection size: 100_000. Hash limit: 100_000. The big one. Largest collection with a perfect hash function.
_000集合大小:100。散列限制:100 _000。最大的一个。具有完美哈希函数的最大集合。
Some surprising stuff here. An initial capacity with 50% additional room at a load factor of 1 wins.
一些令人惊讶的东西。初始容量增加50%,负载系数为1的房间将获胜。
Alright, that's it for the puts. Now, we'll check the gets. Remember, the below maps are all relative to best/worst get times, the put times are no longer taken into account.
好了,这就是看跌期权。现在,我们来检查一下。记住,下面的映射都是相对于最佳/最差的获取时间,放置时间不再被考虑。
Get results
得到结果
Collection size: 100. Hash limit: 50. This means each hash code should occur twice and every other key was expected to collide in the hash map.
集合大小:100。散列限制:50。这意味着每个哈希代码应该出现两次,而其他每个键都应该在哈希映射中发生冲突。
Eh... What?
呃……什么?
Collection size: 100. Hash limit: 90. One in ten keys has a duplicate hash code.
集合大小:100。散列限制:90。每十个键中就有一个有重复的哈希码。
Whoa Nelly! This is the most likely scenario to correlate with the asker's question, and apparently an initial capacity of 100 with a load factor of 1 is one of the worst things here! I swear I didn't fake this.
哇耐莉!这是最可能与asker的问题相关联的场景,显然初始容量为100,负载系数为1是这里最糟糕的事情之一!我发誓我没骗你。
Collection size: 100. Hash limit: 100. Each key as its own unique hash code. No collisions expected.
集合大小:100。散列限制:100。每个键作为它自己的唯一哈希代码。没有碰撞的预期。
This looks a bit more peaceful. Mostly the same results across the board.
这看起来比较平静。基本上所有的结果都是一样的。
Collection size: 1000. Hash limit: 500. Just like in the first test, there's a hash overload of 2 to 1, but now with a lot more entries.
集合大小:1000。散列限制:500。就像第一个测试一样,有一个2到1的哈希重载,但是现在有更多的条目。
Looks like any setting will yield a decent result here.
看起来在这里任何设置都会产生一个不错的结果。
Collection size: 1000. Hash limit: 900. This means one in ten hash codes will occur twice. Reasonable scenario regarding collisions.
集合大小:1000。散列限制:900。这意味着每十个哈希码中就有一个会出现两次。合理的场景中关于碰撞。
And just like with the puts for this setup, we get an anomaly in a strange spot.
就像这个设置一样,我们在一个奇怪的地方得到了一个异常。
Collection size: 1000. Hash limit: 990. Some collisions, but only a few. Quite realistic in this respect.
集合大小:1000。散列限制:990。有些碰撞,但只有几个。在这方面相当现实。
Decent performance everywhere, save for the combination of a high initial capacity with a low load factor. I'd expect this for the puts, since two hash map resizes might be expected. But why on the gets?
所有地方的性能都很好,除了高初始容量和低负载因数。我希望对put这样,因为可能需要两个散列映射大小。但为什么是在获取?
Collection size: 1000. Hash limit: 1000. No duplicate hash codes, but now with a sample size of 1000.
集合大小:1000。散列限制:1000。没有重复的哈希代码,但是现在示例大小为1000。
A wholly unspectacular visualization. This seems to work no matter what.
一个完全不引人注目的可视化。无论如何,这似乎是可行的。
Collection size: 100_000. Hash limit: 10_000. Going into the 100K again, with a whole lot of hash code overlap.
_000集合大小:100。散列限制:10 _000。再次进入100K,有很多哈希代码重叠。
It doesn't look pretty, although the bad spots are very localized. Performance here seems to depend largely on a certain synergy between settings.
它看起来不漂亮,尽管坏点是局部性的。这里的性能似乎主要取决于设置之间的某种协同作用。
Collection size: 100_000. Hash limit: 90_000. A bit more realistic than the previous test, here we've got a 10% overload in hash codes.
_000集合大小:100。散列限制:90 _000。比之前的测试更实际一点,这里我们有10%的哈希代码过载。
Much variance, although if you squint you can see an arrow pointing to the upper right corner.
方差很大,但如果你斜视,你可以看到一个箭头指向右上角。
Collection size: 100_000. Hash limit: 99_000. Good scenario, this. A large collection with a 1% hash code overload.
_000集合大小:100。散列限制:99 _000。良好的情况下,这个问题。包含1%哈希代码过载的大型集合。
Very chaotic. It's hard to find much structure here.
非常混乱。在这里很难找到很多结构。
Collection size: 100_000. Hash limit: 100_000. The big one. Largest collection with a perfect hash function.
_000集合大小:100。散列限制:100 _000。最大的一个。具有完美哈希函数的最大集合。
Anyone else thinks this is starting to look like Atari graphics? This seems to favour an initial capacity of exactly the collection size, -25% or +50%.
有人认为这开始看起来像雅达利图形吗?这似乎有利于初始容量精确的收集大小,-25%或+50%。
Alright, it's time for conclusions now...
好了,现在是结论的时候了…
- Regarding put times: you'll wish to avoid initial capacities that are lower than the expected number of map entries. If an exact number is known beforehand, that number or something slightly above it seems to work best. High load factors can offset lower initial capacities due to earlier hash map resizes. For higher initial capacities, they don't seem to matter that much.
- 关于放置时间:您希望避免初始容量低于预期的map条目数量。如果事先知道了确切的数字,那么这个数字或略高于这个数字的东西似乎最有效。高负载因素可以抵消较早哈希映射调整的低初始容量。对于更高的初始容量,它们似乎没有那么重要。
- Regarding get times: results are slightly chaotic here. There's not much to conclude. It seems to rely very much on subtle ratios between hash code overlap, initial capacity and load factor, with some supposedly bad setups performing well and good setups performing awfully.
- 关于获取时间:这里的结果有点混乱。没什么可下结论的。它似乎非常依赖于散列代码重叠、初始容量和负载因素之间的微妙比率,有些可能糟糕的设置表现良好,而良好的设置表现得非常糟糕。
- I'm apparently full of crap when it comes to assumptions about Java performance. The truth is, unless you are perfectly tuning your settings to the implementation of
HashMap
, the results are going to be all over the place. If there's one thing to take away from this, it's that the default initial size of 16 is a bit dumb for anything but the smallest maps, so use a constructor that sets the initial size if you have any sort of idea about what order of size it's going to be. - 当涉及到对Java性能的假设时,我显然是满嘴废话。事实是,除非您将设置完美地调优为HashMap的实现,否则结果将到处都是。如果有一件事要远离这一点,那就是默认的初始大小16有点愚蠢的最小的地图,所以使用一个构造函数,设置初始大小,如果你有任何想法关于大小的顺序是什么。
- We're measuring in nanoseconds here. The best average time per 10 puts was 1179 ns and the worst 5105 ns on my machine. The best average time per 10 gets was 547 ns and the worst 3484 ns. That may be a factor 6 difference, but we're talking less than a millisecond. On collections that are vastly larger than what the original poster had in mind.
- 我们用纳秒来测量。每10次投中的最佳平均时间是1179纳秒,最坏的是5105纳秒。每10次获得的最佳平均时间是547纳秒,最差的是3484纳秒。这可能是因子6的差异,但我们说的是小于一毫秒。在那些远远超过最初海报的收藏上。
Well, that's it. I hope my code doesn't have some horrendous oversight that invalidates everything I've posted here. This has been fun, and I've learned that in the end you may just as well rely on Java to do its job than to expect much difference from tiny optimizations. That is not to say that some stuff shouldn't be avoided, but then we're mostly talking about constructing lengthy Strings in for loops, using the wrong datastructures and making O(n^3) algorithsm.
嗯,就是这样。我希望我的代码不会有一些可怕的疏忽,使我在这里发布的所有内容无效。这很有趣,而且我还了解到,与期望小优化带来的巨大差异相比,最终您可以依赖Java来完成它的工作。这并不是说,有些东西不能避免,但我们主要讨论建设长字符串在for循环,使用错误的数据结构,使O(n ^ 3)algorithsm。
#2
9
This is a pretty great thread, except there is one crucial thing you're missing. You said:
这是一条非常棒的线,除了有一件重要的事情是你错过的。你说:
Curiously, capacity, capacity+1, capacity+2, capacity-1 and even capacity-10 all yield exactly the same results. I would expect at least capacity-1 and capacity-10 to give worse results.
奇怪的是,容量、容量+1、容量+2、容量-1甚至容量-10都会产生完全相同的结果。我预计至少能力-1和能力-10会产生更坏的结果。
The source code jumps initial capacity the next highest power-of-two internally. That means that, for example, initial capacities of 513, 600, 700, 800, 900, 1000, and 1024 will all use the same initial capacity (1024). This doesn't invalidate the testing done by @G_H though, one should realize that this is being done before analyzing his results. And it does explain the odd behavior of some of the tests.
源代码在内部跳过了初始容量的下一个最高的2次方。这意味着,例如,513,600、700、800、900、1000和1024的初始容量都将使用相同的初始容量(1024)。这并不能使@G_H所做的测试无效,但是在分析结果之前应该意识到这一点。它确实解释了一些测试的奇怪行为。
This is the constructor right for the JDK source:
这是JDK源代码的构造函数:
/**
* Constructs an empty <tt>HashMap</tt> with the specified initial
* capacity and load factor.
*
* @param initialCapacity the initial capacity
* @param loadFactor the load factor
* @throws IllegalArgumentException if the initial capacity is negative
* or the load factor is nonpositive
*/
public HashMap(int initialCapacity, float loadFactor) {
if (initialCapacity < 0)
throw new IllegalArgumentException("Illegal initial capacity: " +
initialCapacity);
if (initialCapacity > MAXIMUM_CAPACITY)
initialCapacity = MAXIMUM_CAPACITY;
if (loadFactor <= 0 || Float.isNaN(loadFactor))
throw new IllegalArgumentException("Illegal load factor: " +
loadFactor);
// Find a power of 2 >= initialCapacity
int capacity = 1;
while (capacity < initialCapacity)
capacity <<= 1;
this.loadFactor = loadFactor;
threshold = (int)(capacity * loadFactor);
table = new Entry[capacity];
init();
}
#3
2
Just go with 101
. I'm not actually sure that it's needed, but it couldn't possibly be worth the effort to ever bother finding out for sure.
101就好。实际上我不确定是否需要它,但它不可能值得花精力去确定。
...just add the 1
.
…就加1。
EDIT: Some justification for my answer.
编辑:我的回答有理由。
First, I'm assuming that your HashMap
will not grow beyond 100
; if it does, you should leave the load-factor as it is. Similarly, if your concern is performance, leave the load-factor as is. If your concern is memory, you can save some by setting the static size. This might maybe be worth doing if you're cramming a lot of stuff in memory; i.e., are storing many maps, or creating heap-space-stressing-sized maps.
首先,我假设您的HashMap不会超过100;如果是这样,您应该将负载因素保留原样。类似地,如果您关心的是性能,那么请保持负载因数不变。如果您关心的是内存,可以通过设置静态大小来保存。如果你在记忆中塞满了很多东西,这可能是值得做的;即。,存储许多地图,或创建堆空间大小的地图。
Second, I choose the value 101
because it offers better readability... if I'm looking at your code afterwards and see that you've set the initial capacity to 100
and you're loading it with 100
elements, I'm going to have to read through the Javadoc to make sure that it won't resize when it reaches precisely 100
. Of course, I won't find the answer there, so I'll have to look at the source. This is not worth it... just leave it 101
and everyone is happy and no-one is looking though the source-code of java.util.HashMap
. Hoorah.
其次,我选择了value 101,因为它提供了更好的可读性……如果我在之后查看您的代码并看到您已经将初始容量设置为100,并且您正在装载100个元素,那么我将必须通读Javadoc,以确保它在达到100时不会调整大小。当然,我不会在这里找到答案,所以我得看一下来源。这不值得……只要把它放在101,每个人都很高兴,没有人会去查看java.util.HashMap的源代码。Hoorah。
Third, the claim that setting the HashMap
to the exact capacity of what you expect with a load factor of 1
"will kill your lookup and insertion performance" is just not true, even if it's made in bold.
第三,将HashMap设置为您所期望的负载因数为1的确切容量“将破坏查找和插入性能”的说法是不正确的,即使它是加粗的。
...if you have n
buckets, and you randomly assign n
items into n
buckets, yep, you're going to end up with items in the same bucket, sure... but that's not the end of the world... in practice, it's just a couple more equals comparisons. In fact, there's esp. little difference when you consider that the alternative is assigning n
items into n/0.75
buckets.
…如果你有n个桶,你随机地将n个项目分配到n个桶中,是的,你最终会得到相同桶中的项目,当然……但这还不是世界末日……在实践中,这只是更多的对等比较。事实上,当你考虑把n个项目分配到n/0.75个桶中时,差别很小。
No need to take my word for it...
没必要相信我的话……
Quick test code:
快速测试代码:
static Random r = new Random();
public static void main(String[] args){
int[] tests = {100, 1000, 10000};
int runs = 5000;
float lf_sta = 1f;
float lf_dyn = 0.75f;
for(int t:tests){
System.err.println("=======Test Put "+t+"");
HashMap<Integer,Integer> map = new HashMap<Integer,Integer>();
long norm_put = testInserts(map, t, runs);
System.err.print("Norm put:"+norm_put+" ms. ");
int cap_sta = t;
map = new HashMap<Integer,Integer>(cap_sta, lf_sta);
long sta_put = testInserts(map, t, runs);
System.err.print("Static put:"+sta_put+" ms. ");
int cap_dyn = (int)Math.ceil((float)t/lf_dyn);
map = new HashMap<Integer,Integer>(cap_dyn, lf_dyn);
long dyn_put = testInserts(map, t, runs);
System.err.println("Dynamic put:"+dyn_put+" ms. ");
}
for(int t:tests){
System.err.println("=======Test Get (hits) "+t+"");
HashMap<Integer,Integer> map = new HashMap<Integer,Integer>();
fill(map, t);
long norm_get_hits = testGetHits(map, t, runs);
System.err.print("Norm get (hits):"+norm_get_hits+" ms. ");
int cap_sta = t;
map = new HashMap<Integer,Integer>(cap_sta, lf_sta);
fill(map, t);
long sta_get_hits = testGetHits(map, t, runs);
System.err.print("Static get (hits):"+sta_get_hits+" ms. ");
int cap_dyn = (int)Math.ceil((float)t/lf_dyn);
map = new HashMap<Integer,Integer>(cap_dyn, lf_dyn);
fill(map, t);
long dyn_get_hits = testGetHits(map, t, runs);
System.err.println("Dynamic get (hits):"+dyn_get_hits+" ms. ");
}
for(int t:tests){
System.err.println("=======Test Get (Rand) "+t+"");
HashMap<Integer,Integer> map = new HashMap<Integer,Integer>();
fill(map, t);
long norm_get_rand = testGetRand(map, t, runs);
System.err.print("Norm get (rand):"+norm_get_rand+" ms. ");
int cap_sta = t;
map = new HashMap<Integer,Integer>(cap_sta, lf_sta);
fill(map, t);
long sta_get_rand = testGetRand(map, t, runs);
System.err.print("Static get (rand):"+sta_get_rand+" ms. ");
int cap_dyn = (int)Math.ceil((float)t/lf_dyn);
map = new HashMap<Integer,Integer>(cap_dyn, lf_dyn);
fill(map, t);
long dyn_get_rand = testGetRand(map, t, runs);
System.err.println("Dynamic get (rand):"+dyn_get_rand+" ms. ");
}
}
public static long testInserts(HashMap<Integer,Integer> map, int test, int runs){
long b4 = System.currentTimeMillis();
for(int i=0; i<runs; i++){
fill(map, test);
map.clear();
}
return System.currentTimeMillis()-b4;
}
public static void fill(HashMap<Integer,Integer> map, int test){
for(int j=0; j<test; j++){
if(map.put(r.nextInt(), j)!=null){
j--;
}
}
}
public static long testGetHits(HashMap<Integer,Integer> map, int test, int runs){
long b4 = System.currentTimeMillis();
ArrayList<Integer> keys = new ArrayList<Integer>();
keys.addAll(map.keySet());
for(int i=0; i<runs; i++){
for(int j=0; j<test; j++){
keys.get(r.nextInt(keys.size()));
}
}
return System.currentTimeMillis()-b4;
}
public static long testGetRand(HashMap<Integer,Integer> map, int test, int runs){
long b4 = System.currentTimeMillis();
for(int i=0; i<runs; i++){
for(int j=0; j<test; j++){
map.get(r.nextInt());
}
}
return System.currentTimeMillis()-b4;
}
Test Results:
测试结果:
=======Test Put 100
Norm put:78 ms. Static put:78 ms. Dynamic put:62 ms.
=======Test Put 1000
Norm put:764 ms. Static put:763 ms. Dynamic put:748 ms.
=======Test Put 10000
Norm put:12921 ms. Static put:12889 ms. Dynamic put:12873 ms.
=======Test Get (hits) 100
Norm get (hits):47 ms. Static get (hits):31 ms. Dynamic get (hits):32 ms.
=======Test Get (hits) 1000
Norm get (hits):327 ms. Static get (hits):328 ms. Dynamic get (hits):343 ms.
=======Test Get (hits) 10000
Norm get (hits):3304 ms. Static get (hits):3366 ms. Dynamic get (hits):3413 ms.
=======Test Get (Rand) 100
Norm get (rand):63 ms. Static get (rand):46 ms. Dynamic get (rand):47 ms.
=======Test Get (Rand) 1000
Norm get (rand):483 ms. Static get (rand):499 ms. Dynamic get (rand):483 ms.
=======Test Get (Rand) 10000
Norm get (rand):5190 ms. Static get (rand):5362 ms. Dynamic get (rand):5236 ms.
re: ↑ — there's about this →||← much difference between the different settings.
re:↑——有这个→| |←区别不同的设置。
With respect to my original answer (the bit above the first horizontal line), it was deliberately glib because in most cases, this type of micro-optimising is not good.
对于我最初的答案(在第一个水平线上面的位),它是故意的,因为在大多数情况下,这种类型的微优化是不好的。
#4
1
From the HashMap
JavaDoc:
从HashMap JavaDoc:
As a general rule, the default load factor (.75) offers a good tradeoff between time and space costs. Higher values decrease the space overhead but increase the lookup cost (reflected in most of the operations of the HashMap class, including get and put). The expected number of entries in the map and its load factor should be taken into account when setting its initial capacity, so as to minimize the number of rehash operations. If the initial capacity is greater than the maximum number of entries divided by the load factor, no rehash operations will ever occur.
一般来说,默认负载因数(.75)在时间和空间成本之间提供了很好的权衡。更高的值减少了空间开销,但增加了查找成本(反映在HashMap类的大多数操作中,包括get和put)。在设置映射的初始容量时,应该考虑映射中预期的条目数量及其负载因数,以最小化rehash操作的数量。如果初始容量大于按负载因素划分的最大条目数,则不会发生重新散列操作。
So if you're expecting 100 entries, perhaps a load factor of 0.75 and an initial capacity of ceiling(100/0.75) would be best. That comes down to 134.
因此,如果您预期有100个条目,那么负载因数为0.75和初始容量上限(100/0.75)将是最好的。下降到134。
I have to admit, I'm not certain why lookup cost would be greater for a higher load factor. Just because the HashMap is more "crowded" doesn't mean that more objects will be placed in the same bucket, right? That only depends on their hash code, if I'm not mistaken. So assuming a decent hash code spread, shouldn't most cases still be O(1) regardless of load factor?
我不得不承认,我不确定为什么查找的成本会更高。仅仅因为HashMap更“拥挤”并不意味着将有更多的对象放在同一个bucket中,对吗?这只取决于他们的哈希代码,如果我没弄错的话。假设哈希代码分布良好,那么不管负载因素如何,大多数情况下不应该仍然是O(1)吗?
EDIT: I should read more before posting... Of course the hash code cannot directly map to some internal index. It must be reduced to a value that fits the current capacity. Meaning that the greater your initial capacity, the smaller you can expect the number of hash collisions to be. Choosing an initial capacity exactly the size (or +1) of your object set with a load factor of 1 will indeed make sure that your map is never resized. However, it will kill your lookup and insertion performance. A resize is still relatively quick and would only occur maybe once, while lookups are done on pretty much any relevant work with the map. As a result, optimizing for quick lookups is what you really want here. You can combine that with never having to resize by doing as the JavaDoc says: take your required capacity, divide by an optimal load factor (e.g. 0.75) and use that as the initial capacity, with that load factor. Add 1 to make sure rounding doesn't get you.
编辑:我应该在发布之前多读一些……当然,哈希代码不能直接映射到某个内部索引。它必须被降低到适合当前容量的值。也就是说,初始容量越大,哈希冲突的数量就越小。选择一个初始容量(或+1)对象集的大小(负载系数为1)确实可以确保映射不会被调整大小。但是,它会破坏查找和插入性能。调整大小仍然相对较快,可能只会发生一次,而查找几乎都是在与映射相关的工作上进行的。因此,优化快速查找是您在这里真正需要的。您可以将其与不需要调整大小结合起来,如JavaDoc所言:取所需的容量,除以最优负载因子(例如0.75),然后使用该负载因子作为初始容量。加1以确保四舍五入没有得到。
#1
66
Alright, to put this thing to rest, I've created a test app to run a couple of scenarios and get some visualizations of the results. Here's how the tests are done:
好了,为了让这个东西停下来,我创建了一个测试应用程序来运行一些场景并得到结果的可视化效果。下面是测试的方法:
- A number of different collection sizes have been tried: one hundred, one thousand and one hundred thousand entries.
- 已经尝试了许多不同的收集尺寸:100、1000和10万条目。
- The keys used are instances of a class that are uniquely identified by an ID. Each test uses unique keys, with incrementing integers as IDs. The
equals
method only uses the ID, so no key mapping overwrites another one. - 使用的键是由ID唯一标识的类的实例。每个测试使用唯一的键,递增的整数作为ID。equals方法只使用ID,因此没有键映射重写另一个ID。
- The keys get a hash code that consists of the module remainder of their ID against some preset number. We'll call that number the hash limit. This allowed me to control the number of hash collisions that would be expected. For example, if our collection size is 100, we'll have keys with IDs ranging from 0 to 99. If the hash limit is 100, every key will have a unique hash code. If the hash limit is 50, key 0 will have the same hash code as key 50, 1 will have the same hash code as 51 etc. In other words, the expected number of hash collisions per key is the collection size divided by the hash limit.
- 键获取一个散列代码,该散列代码由它们的ID的模块余数与某个预设值组成。我们称之为哈希极限。这使我能够控制预期的散列冲突的数量。例如,如果我们的集合大小是100,那么我们将有id从0到99的键。如果哈希限制是100,每个键都有一个唯一的哈希代码。如果哈希限制是50,那么key 0将具有与key 50相同的哈希代码,1将具有与51相同的哈希代码,等等。换句话说,每个键的哈希冲突的期望值是集合大小除以哈希限制。
- For each combination of collection size and hash limit, I've run the test using hash maps initialized with different settings. These settings are the load factor, and an initial capacity that is expressed as a factor of the collection setting. For example, a test with a collection size of 100 and an initial capacity factor of 1.25 will initialize a hash map with an initial capacity of 125.
- 对于集合大小和散列限制的每种组合,我都使用不同设置初始化的散列映射来运行测试。这些设置是负载因数,以及表示为集合设置因数的初始容量。例如,一个集合大小为100,初始容量因子为1.25的测试将初始化一个初始容量为125的散列映射。
- The value for each key is simply a new
Object
. - 每个键的值只是一个新对象。
- Each test result is encapsulated in an instance of a Result class. At the end of all tests, the results are ordered from worst overall performance to best.
- 每个测试结果都封装在一个result类的实例中。在所有测试的最后,结果都是从最糟糕的整体性能到最好的。
- The average time for puts and gets is calculated per 10 puts/gets.
- 投入和获得的平均时间是每10次投入/获得。
- All test combinations are run once to eliminate JIT compilation influence. After that, the tests are run for actual results.
- 所有的测试组合都运行一次,以消除JIT编译的影响。之后,将运行测试以获得实际结果。
Here's the class:
类:
package hashmaptest;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
public class HashMapTest {
private static final List<Result> results = new ArrayList<Result>();
public static void main(String[] args) throws IOException {
//First entry of each array is the sample collection size, subsequent entries
//are the hash limits
final int[][] sampleSizesAndHashLimits = new int[][] {
{100, 50, 90, 100},
{1000, 500, 900, 990, 1000},
{100000, 10000, 90000, 99000, 100000}
};
final double[] initialCapacityFactors = new double[] {0.5, 0.75, 1.0, 1.25, 1.5, 2.0};
final float[] loadFactors = new float[] {0.5f, 0.75f, 1.0f, 1.25f};
//Doing a warmup run to eliminate JIT influence
for(int[] sizeAndLimits : sampleSizesAndHashLimits) {
int size = sizeAndLimits[0];
for(int i = 1; i < sizeAndLimits.length; ++i) {
int limit = sizeAndLimits[i];
for(double initCapacityFactor : initialCapacityFactors) {
for(float loadFactor : loadFactors) {
runTest(limit, size, initCapacityFactor, loadFactor);
}
}
}
}
results.clear();
//Now for the real thing...
for(int[] sizeAndLimits : sampleSizesAndHashLimits) {
int size = sizeAndLimits[0];
for(int i = 1; i < sizeAndLimits.length; ++i) {
int limit = sizeAndLimits[i];
for(double initCapacityFactor : initialCapacityFactors) {
for(float loadFactor : loadFactors) {
runTest(limit, size, initCapacityFactor, loadFactor);
}
}
}
}
Collections.sort(results);
for(final Result result : results) {
result.printSummary();
}
// ResultVisualizer.visualizeResults(results);
}
private static void runTest(final int hashLimit, final int sampleSize,
final double initCapacityFactor, final float loadFactor) {
final int initialCapacity = (int)(sampleSize * initCapacityFactor);
System.out.println("Running test for a sample collection of size " + sampleSize
+ ", an initial capacity of " + initialCapacity + ", a load factor of "
+ loadFactor + " and keys with a hash code limited to " + hashLimit);
System.out.println("====================");
double hashOverload = (((double)sampleSize/hashLimit) - 1.0) * 100.0;
System.out.println("Hash code overload: " + hashOverload + "%");
//Generating our sample key collection.
final List<Key> keys = generateSamples(hashLimit, sampleSize);
//Generating our value collection
final List<Object> values = generateValues(sampleSize);
final HashMap<Key, Object> map = new HashMap<Key, Object>(initialCapacity, loadFactor);
final long startPut = System.nanoTime();
for(int i = 0; i < sampleSize; ++i) {
map.put(keys.get(i), values.get(i));
}
final long endPut = System.nanoTime();
final long putTime = endPut - startPut;
final long averagePutTime = putTime/(sampleSize/10);
System.out.println("Time to map all keys to their values: " + putTime + " ns");
System.out.println("Average put time per 10 entries: " + averagePutTime + " ns");
final long startGet = System.nanoTime();
for(int i = 0; i < sampleSize; ++i) {
map.get(keys.get(i));
}
final long endGet = System.nanoTime();
final long getTime = endGet - startGet;
final long averageGetTime = getTime/(sampleSize/10);
System.out.println("Time to get the value for every key: " + getTime + " ns");
System.out.println("Average get time per 10 entries: " + averageGetTime + " ns");
System.out.println("");
final Result result =
new Result(sampleSize, initialCapacity, loadFactor, hashOverload, averagePutTime, averageGetTime, hashLimit);
results.add(result);
//Haha, what kind of noob explicitly calls for garbage collection?
System.gc();
try {
Thread.sleep(200);
} catch(final InterruptedException e) {}
}
private static List<Key> generateSamples(final int hashLimit, final int sampleSize) {
final ArrayList<Key> result = new ArrayList<Key>(sampleSize);
for(int i = 0; i < sampleSize; ++i) {
result.add(new Key(i, hashLimit));
}
return result;
}
private static List<Object> generateValues(final int sampleSize) {
final ArrayList<Object> result = new ArrayList<Object>(sampleSize);
for(int i = 0; i < sampleSize; ++i) {
result.add(new Object());
}
return result;
}
private static class Key {
private final int hashCode;
private final int id;
Key(final int id, final int hashLimit) {
//Equals implies same hashCode if limit is the same
//Same hashCode doesn't necessarily implies equals
this.id = id;
this.hashCode = id % hashLimit;
}
@Override
public int hashCode() {
return hashCode;
}
@Override
public boolean equals(final Object o) {
return ((Key)o).id == this.id;
}
}
static class Result implements Comparable<Result> {
final int sampleSize;
final int initialCapacity;
final float loadFactor;
final double hashOverloadPercentage;
final long averagePutTime;
final long averageGetTime;
final int hashLimit;
Result(final int sampleSize, final int initialCapacity, final float loadFactor,
final double hashOverloadPercentage, final long averagePutTime,
final long averageGetTime, final int hashLimit) {
this.sampleSize = sampleSize;
this.initialCapacity = initialCapacity;
this.loadFactor = loadFactor;
this.hashOverloadPercentage = hashOverloadPercentage;
this.averagePutTime = averagePutTime;
this.averageGetTime = averageGetTime;
this.hashLimit = hashLimit;
}
@Override
public int compareTo(final Result o) {
final long putDiff = o.averagePutTime - this.averagePutTime;
final long getDiff = o.averageGetTime - this.averageGetTime;
return (int)(putDiff + getDiff);
}
void printSummary() {
System.out.println("" + averagePutTime + " ns per 10 puts, "
+ averageGetTime + " ns per 10 gets, for a load factor of "
+ loadFactor + ", initial capacity of " + initialCapacity
+ " for " + sampleSize + " mappings and " + hashOverloadPercentage
+ "% hash code overload.");
}
}
}
Running this might take a while. The results are printed out on standard out. You might notice I've commented out a line. That line calls a visualizer that outputs visual representations of the results to png files. The class for this is given below. If you wish to run it, uncomment the appropriate line in the code above. Be warned: the visualizer class assumes you're running on Windows and will create folders and files in C:\temp. When running on another platform, adjust this.
运行这个可能需要一段时间。结果印在标准输出上。你可能注意到我注释了一行。这一行调用了一个visualizer,它将结果的可视化表示输出到png文件中。这方面的类如下所示。如果您希望运行它,请取消上面代码中适当的行注释。注意:visualizer类假定您在Windows上运行,并将在C:\temp中创建文件夹和文件。在另一个平台上运行时,请对此进行调整。
package hashmaptest;
import hashmaptest.HashMapTest.Result;
import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.text.DecimalFormat;
import java.text.NumberFormat;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import javax.imageio.ImageIO;
public class ResultVisualizer {
private static final Map<Integer, Map<Integer, Set<Result>>> sampleSizeToHashLimit =
new HashMap<Integer, Map<Integer, Set<Result>>>();
private static final DecimalFormat df = new DecimalFormat("0.00");
static void visualizeResults(final List<Result> results) throws IOException {
final File tempFolder = new File("C:\\temp");
final File baseFolder = makeFolder(tempFolder, "hashmap_tests");
long bestPutTime = -1L;
long worstPutTime = 0L;
long bestGetTime = -1L;
long worstGetTime = 0L;
for(final Result result : results) {
final Integer sampleSize = result.sampleSize;
final Integer hashLimit = result.hashLimit;
final long putTime = result.averagePutTime;
final long getTime = result.averageGetTime;
if(bestPutTime == -1L || putTime < bestPutTime)
bestPutTime = putTime;
if(bestGetTime <= -1.0f || getTime < bestGetTime)
bestGetTime = getTime;
if(putTime > worstPutTime)
worstPutTime = putTime;
if(getTime > worstGetTime)
worstGetTime = getTime;
Map<Integer, Set<Result>> hashLimitToResults =
sampleSizeToHashLimit.get(sampleSize);
if(hashLimitToResults == null) {
hashLimitToResults = new HashMap<Integer, Set<Result>>();
sampleSizeToHashLimit.put(sampleSize, hashLimitToResults);
}
Set<Result> resultSet = hashLimitToResults.get(hashLimit);
if(resultSet == null) {
resultSet = new HashSet<Result>();
hashLimitToResults.put(hashLimit, resultSet);
}
resultSet.add(result);
}
System.out.println("Best average put time: " + bestPutTime + " ns");
System.out.println("Best average get time: " + bestGetTime + " ns");
System.out.println("Worst average put time: " + worstPutTime + " ns");
System.out.println("Worst average get time: " + worstGetTime + " ns");
for(final Integer sampleSize : sampleSizeToHashLimit.keySet()) {
final File sizeFolder = makeFolder(baseFolder, "sample_size_" + sampleSize);
final Map<Integer, Set<Result>> hashLimitToResults =
sampleSizeToHashLimit.get(sampleSize);
for(final Integer hashLimit : hashLimitToResults.keySet()) {
final File limitFolder = makeFolder(sizeFolder, "hash_limit_" + hashLimit);
final Set<Result> resultSet = hashLimitToResults.get(hashLimit);
final Set<Float> loadFactorSet = new HashSet<Float>();
final Set<Integer> initialCapacitySet = new HashSet<Integer>();
for(final Result result : resultSet) {
loadFactorSet.add(result.loadFactor);
initialCapacitySet.add(result.initialCapacity);
}
final List<Float> loadFactors = new ArrayList<Float>(loadFactorSet);
final List<Integer> initialCapacities = new ArrayList<Integer>(initialCapacitySet);
Collections.sort(loadFactors);
Collections.sort(initialCapacities);
final BufferedImage putImage =
renderMap(resultSet, loadFactors, initialCapacities, worstPutTime, bestPutTime, false);
final BufferedImage getImage =
renderMap(resultSet, loadFactors, initialCapacities, worstGetTime, bestGetTime, true);
final String putFileName = "size_" + sampleSize + "_hlimit_" + hashLimit + "_puts.png";
final String getFileName = "size_" + sampleSize + "_hlimit_" + hashLimit + "_gets.png";
writeImage(putImage, limitFolder, putFileName);
writeImage(getImage, limitFolder, getFileName);
}
}
}
private static File makeFolder(final File parent, final String folder) throws IOException {
final File child = new File(parent, folder);
if(!child.exists())
child.mkdir();
return child;
}
private static BufferedImage renderMap(final Set<Result> results, final List<Float> loadFactors,
final List<Integer> initialCapacities, final float worst, final float best,
final boolean get) {
//[x][y] => x is mapped to initial capacity, y is mapped to load factor
final Color[][] map = new Color[initialCapacities.size()][loadFactors.size()];
for(final Result result : results) {
final int x = initialCapacities.indexOf(result.initialCapacity);
final int y = loadFactors.indexOf(result.loadFactor);
final float time = get ? result.averageGetTime : result.averagePutTime;
final float score = (time - best)/(worst - best);
final Color c = new Color(score, 1.0f - score, 0.0f);
map[x][y] = c;
}
final int imageWidth = initialCapacities.size() * 40 + 50;
final int imageHeight = loadFactors.size() * 40 + 50;
final BufferedImage image =
new BufferedImage(imageWidth, imageHeight, BufferedImage.TYPE_3BYTE_BGR);
final Graphics2D g = image.createGraphics();
g.setColor(Color.WHITE);
g.fillRect(0, 0, imageWidth, imageHeight);
for(int x = 0; x < map.length; ++x) {
for(int y = 0; y < map[x].length; ++y) {
g.setColor(map[x][y]);
g.fillRect(50 + x*40, imageHeight - 50 - (y+1)*40, 40, 40);
g.setColor(Color.BLACK);
g.drawLine(25, imageHeight - 50 - (y+1)*40, 50, imageHeight - 50 - (y+1)*40);
final Float loadFactor = loadFactors.get(y);
g.drawString(df.format(loadFactor), 10, imageHeight - 65 - (y)*40);
}
g.setColor(Color.BLACK);
g.drawLine(50 + (x+1)*40, imageHeight - 50, 50 + (x+1)*40, imageHeight - 15);
final int initialCapacity = initialCapacities.get(x);
g.drawString(((initialCapacity%1000 == 0) ? "" + (initialCapacity/1000) + "K" : "" + initialCapacity), 15 + (x+1)*40, imageHeight - 25);
}
g.drawLine(25, imageHeight - 50, imageWidth, imageHeight - 50);
g.drawLine(50, 0, 50, imageHeight - 25);
g.dispose();
return image;
}
private static void writeImage(final BufferedImage image, final File folder,
final String filename) throws IOException {
final File imageFile = new File(folder, filename);
ImageIO.write(image, "png", imageFile);
}
}
The visualized output is as follows:
可视化输出如下:
- Tests are divided first by collection size, then by hash limit.
- 测试首先要除以集合大小,然后再除以散列限制。
- For each test, there's an output image regarding the average put time (per 10 puts) and the average get time (per 10 gets). The images are two-dimensional "heat maps" that show a color per combination of initial capacity and load factor.
- 对于每个测试,都有一个输出映像,关于平均put时间(每10次put)和平均get时间(每10次get)。这些图像是二维的“热图”,显示了初始容量和负载系数的组合颜色。
- The colours in the images are based on the average time on a normalized scale from best to worst result, ranging from saturated green to saturated red. In other words, the best time will be fully green, while the worst time will be fully red. Two different time measurements should never have the same colour.
- 图像中的颜色是基于标准化尺度上的平均时间,从最佳到最差的结果,从饱和的绿色到饱和的红色不等。换句话说,最好的时间是完全绿色的,最坏的时间是完全红色的。两个不同的时间测量不应该有相同的颜色。
- The colour maps are calculated separately for puts and gets, but encompass all tests for their respective categories.
- 颜色地图分别为put和get计算,但包含所有针对它们各自类别的测试。
- The visualizations show the initial capacity on their x axis, and the load factor on the y axis.
- 可视化显示了它们x轴上的初始容量,以及y轴上的负载因子。
Without further ado, let's take a look at the results. I'll start with the results for puts.
不用多说了,我们来看一下结果。我将从put的结果开始。
Put results
把结果
Collection size: 100. Hash limit: 50. This means each hash code should occur twice and every other key collides in the hash map.
集合大小:100。散列限制:50。这意味着每个哈希代码应该出现两次,而哈希映射中的每个其他键都发生冲突。
Well, that doesn't start off very good. We see that there's a big hotspot for an initial capacity 25% above the collection size, with a load factor of 1. The lower left corner doesn't perform too well.
这并不是很好开始。我们看到有一个较大的热点,初始容量比集合大小高出25%,负载系数为1。左下角的表现不太好。
Collection size: 100. Hash limit: 90. One in ten keys has a duplicate hash code.
集合大小:100。散列限制:90。每十个键中就有一个有重复的哈希码。
This is a slightly more realistic scenario, not having a perfect hash function but still 10% overload. The hotspot is gone, but the combination of a low initial capacity with a low load factor obviously doesn't work.
这是一个稍微真实的场景,没有完美的哈希函数,但仍然有10%的超负荷。热点消失了,但低初始容量和低负荷系数的组合显然不起作用。
Collection size: 100. Hash limit: 100. Each key as its own unique hash code. No collisions expected if there are enough buckets.
集合大小:100。散列限制:100。每个键作为它自己的唯一哈希代码。如果有足够的桶,预计不会发生碰撞。
An initial capacity of 100 with a load factor of 1 seems fine. Surprisingly, a higher initial capacity with a lower load factor isn't necessarily good.
初始容量为100的负载系数为1似乎很好。令人惊讶的是,高初始容量和低负荷系数并不一定是好的。
Collection size: 1000. Hash limit: 500. It's getting more serious here, with 1000 entries. Just like in the first test, there's a hash overload of 2 to 1.
集合大小:1000。散列限制:500。这里的问题越来越严重,有1000个条目。就像第一个测试一样,哈希重载是2比1。
The lower left corner is still not doing well. But there seems to be a symmetry between the combo of lower initial count/high load factor and higher initial count/low load factor.
左下角仍然表现不佳。但较低的初始计数/高负荷系数和较高的初始计数/低负荷系数之间似乎存在对称性。
Collection size: 1000. Hash limit: 900. This means one in ten hash codes will occur twice. Reasonable scenario regarding collisions.
集合大小:1000。散列限制:900。这意味着每十个哈希码中就有一个会出现两次。合理的场景中关于碰撞。
There's something very funny going on with the unlikely combo of an initial capacity that's too low with a load factor above 1, which is rather counter-intuitive. Otherwise, still quite symmetrical.
有一件非常有趣的事情发生了,那就是初始容量的组合不太可能是太低,负载系数大于1,这是非常反直觉的。否则,仍然很对称。
Collection size: 1000. Hash limit: 990. Some collisions, but only a few. Quite realistic in this respect.
集合大小:1000。散列限制:990。有些碰撞,但只有几个。在这方面相当现实。
We've got a nice symmetry here. Lower left corner is still sub-optimal, but the combos 1000 init capacity/1.0 load factor versus 1250 init capacity/0.75 load factor are at the same level.
这里有一个很好的对称性。左下角仍然是次优的,但是combos 1000 init容量/1.0负载因数与1250 init容量/0.75负载因数是相同的。
Collection size: 1000. Hash limit: 1000. No duplicate hash codes, but now with a sample size of 1000.
集合大小:1000。散列限制:1000。没有重复的哈希代码,但是现在示例大小为1000。
Not much to be said here. The combination of a higher initial capacity with a load factor of 0.75 seems to slightly outperform the combination of 1000 initial capacity with a load factor of 1.
这里没什么可说的。更高的初始容量和0.75的负载系数的组合似乎稍微优于1000的初始容量和1的负载系数的组合。
Collection size: 100_000. Hash limit: 10_000. Alright, it's getting serious now, with a sample size of one hundred thousand and 100 hash code duplicates per key.
_000集合大小:100。散列限制:10 _000。好了,现在它变得很重要了,每个键的样本大小是10万100个哈希代码重复。
Yikes! I think we found our lower spectrum. An init capacity of exactly the collection size with a load factor of 1 is doing really well here, but other than that it's all over the shop.
呵!我们找到了较低的频谱。加载系数为1的初始化容量在这里做得很好,但除此之外,它在整个商店都有。
Collection size: 100_000. Hash limit: 90_000. A bit more realistic than the previous test, here we've got a 10% overload in hash codes.
_000集合大小:100。散列限制:90 _000。比之前的测试更实际一点,这里我们有10%的哈希代码过载。
The lower left corner is still undesirable. Higher initial capacities work best.
左下角仍然是不可取的。较高的初始能力最好。
Collection size: 100_000. Hash limit: 99_000. Good scenario, this. A large collection with a 1% hash code overload.
_000集合大小:100。散列限制:99 _000。良好的情况下,这个问题。包含1%哈希代码过载的大型集合。
Using the exact collection size as init capacity with a load factor of 1 wins out here! Slightly larger init capacities work quite well, though.
使用准确的集合大小作为initwithload因数为1的init容量在这里胜出!不过,稍微大一点的init容量可以很好地工作。
Collection size: 100_000. Hash limit: 100_000. The big one. Largest collection with a perfect hash function.
_000集合大小:100。散列限制:100 _000。最大的一个。具有完美哈希函数的最大集合。
Some surprising stuff here. An initial capacity with 50% additional room at a load factor of 1 wins.
一些令人惊讶的东西。初始容量增加50%,负载系数为1的房间将获胜。
Alright, that's it for the puts. Now, we'll check the gets. Remember, the below maps are all relative to best/worst get times, the put times are no longer taken into account.
好了,这就是看跌期权。现在,我们来检查一下。记住,下面的映射都是相对于最佳/最差的获取时间,放置时间不再被考虑。
Get results
得到结果
Collection size: 100. Hash limit: 50. This means each hash code should occur twice and every other key was expected to collide in the hash map.
集合大小:100。散列限制:50。这意味着每个哈希代码应该出现两次,而其他每个键都应该在哈希映射中发生冲突。
Eh... What?
呃……什么?
Collection size: 100. Hash limit: 90. One in ten keys has a duplicate hash code.
集合大小:100。散列限制:90。每十个键中就有一个有重复的哈希码。
Whoa Nelly! This is the most likely scenario to correlate with the asker's question, and apparently an initial capacity of 100 with a load factor of 1 is one of the worst things here! I swear I didn't fake this.
哇耐莉!这是最可能与asker的问题相关联的场景,显然初始容量为100,负载系数为1是这里最糟糕的事情之一!我发誓我没骗你。
Collection size: 100. Hash limit: 100. Each key as its own unique hash code. No collisions expected.
集合大小:100。散列限制:100。每个键作为它自己的唯一哈希代码。没有碰撞的预期。
This looks a bit more peaceful. Mostly the same results across the board.
这看起来比较平静。基本上所有的结果都是一样的。
Collection size: 1000. Hash limit: 500. Just like in the first test, there's a hash overload of 2 to 1, but now with a lot more entries.
集合大小:1000。散列限制:500。就像第一个测试一样,有一个2到1的哈希重载,但是现在有更多的条目。
Looks like any setting will yield a decent result here.
看起来在这里任何设置都会产生一个不错的结果。
Collection size: 1000. Hash limit: 900. This means one in ten hash codes will occur twice. Reasonable scenario regarding collisions.
集合大小:1000。散列限制:900。这意味着每十个哈希码中就有一个会出现两次。合理的场景中关于碰撞。
And just like with the puts for this setup, we get an anomaly in a strange spot.
就像这个设置一样,我们在一个奇怪的地方得到了一个异常。
Collection size: 1000. Hash limit: 990. Some collisions, but only a few. Quite realistic in this respect.
集合大小:1000。散列限制:990。有些碰撞,但只有几个。在这方面相当现实。
Decent performance everywhere, save for the combination of a high initial capacity with a low load factor. I'd expect this for the puts, since two hash map resizes might be expected. But why on the gets?
所有地方的性能都很好,除了高初始容量和低负载因数。我希望对put这样,因为可能需要两个散列映射大小。但为什么是在获取?
Collection size: 1000. Hash limit: 1000. No duplicate hash codes, but now with a sample size of 1000.
集合大小:1000。散列限制:1000。没有重复的哈希代码,但是现在示例大小为1000。
A wholly unspectacular visualization. This seems to work no matter what.
一个完全不引人注目的可视化。无论如何,这似乎是可行的。
Collection size: 100_000. Hash limit: 10_000. Going into the 100K again, with a whole lot of hash code overlap.
_000集合大小:100。散列限制:10 _000。再次进入100K,有很多哈希代码重叠。
It doesn't look pretty, although the bad spots are very localized. Performance here seems to depend largely on a certain synergy between settings.
它看起来不漂亮,尽管坏点是局部性的。这里的性能似乎主要取决于设置之间的某种协同作用。
Collection size: 100_000. Hash limit: 90_000. A bit more realistic than the previous test, here we've got a 10% overload in hash codes.
_000集合大小:100。散列限制:90 _000。比之前的测试更实际一点,这里我们有10%的哈希代码过载。
Much variance, although if you squint you can see an arrow pointing to the upper right corner.
方差很大,但如果你斜视,你可以看到一个箭头指向右上角。
Collection size: 100_000. Hash limit: 99_000. Good scenario, this. A large collection with a 1% hash code overload.
_000集合大小:100。散列限制:99 _000。良好的情况下,这个问题。包含1%哈希代码过载的大型集合。
Very chaotic. It's hard to find much structure here.
非常混乱。在这里很难找到很多结构。
Collection size: 100_000. Hash limit: 100_000. The big one. Largest collection with a perfect hash function.
_000集合大小:100。散列限制:100 _000。最大的一个。具有完美哈希函数的最大集合。
Anyone else thinks this is starting to look like Atari graphics? This seems to favour an initial capacity of exactly the collection size, -25% or +50%.
有人认为这开始看起来像雅达利图形吗?这似乎有利于初始容量精确的收集大小,-25%或+50%。
Alright, it's time for conclusions now...
好了,现在是结论的时候了…
- Regarding put times: you'll wish to avoid initial capacities that are lower than the expected number of map entries. If an exact number is known beforehand, that number or something slightly above it seems to work best. High load factors can offset lower initial capacities due to earlier hash map resizes. For higher initial capacities, they don't seem to matter that much.
- 关于放置时间:您希望避免初始容量低于预期的map条目数量。如果事先知道了确切的数字,那么这个数字或略高于这个数字的东西似乎最有效。高负载因素可以抵消较早哈希映射调整的低初始容量。对于更高的初始容量,它们似乎没有那么重要。
- Regarding get times: results are slightly chaotic here. There's not much to conclude. It seems to rely very much on subtle ratios between hash code overlap, initial capacity and load factor, with some supposedly bad setups performing well and good setups performing awfully.
- 关于获取时间:这里的结果有点混乱。没什么可下结论的。它似乎非常依赖于散列代码重叠、初始容量和负载因素之间的微妙比率,有些可能糟糕的设置表现良好,而良好的设置表现得非常糟糕。
- I'm apparently full of crap when it comes to assumptions about Java performance. The truth is, unless you are perfectly tuning your settings to the implementation of
HashMap
, the results are going to be all over the place. If there's one thing to take away from this, it's that the default initial size of 16 is a bit dumb for anything but the smallest maps, so use a constructor that sets the initial size if you have any sort of idea about what order of size it's going to be. - 当涉及到对Java性能的假设时,我显然是满嘴废话。事实是,除非您将设置完美地调优为HashMap的实现,否则结果将到处都是。如果有一件事要远离这一点,那就是默认的初始大小16有点愚蠢的最小的地图,所以使用一个构造函数,设置初始大小,如果你有任何想法关于大小的顺序是什么。
- We're measuring in nanoseconds here. The best average time per 10 puts was 1179 ns and the worst 5105 ns on my machine. The best average time per 10 gets was 547 ns and the worst 3484 ns. That may be a factor 6 difference, but we're talking less than a millisecond. On collections that are vastly larger than what the original poster had in mind.
- 我们用纳秒来测量。每10次投中的最佳平均时间是1179纳秒,最坏的是5105纳秒。每10次获得的最佳平均时间是547纳秒,最差的是3484纳秒。这可能是因子6的差异,但我们说的是小于一毫秒。在那些远远超过最初海报的收藏上。
Well, that's it. I hope my code doesn't have some horrendous oversight that invalidates everything I've posted here. This has been fun, and I've learned that in the end you may just as well rely on Java to do its job than to expect much difference from tiny optimizations. That is not to say that some stuff shouldn't be avoided, but then we're mostly talking about constructing lengthy Strings in for loops, using the wrong datastructures and making O(n^3) algorithsm.
嗯,就是这样。我希望我的代码不会有一些可怕的疏忽,使我在这里发布的所有内容无效。这很有趣,而且我还了解到,与期望小优化带来的巨大差异相比,最终您可以依赖Java来完成它的工作。这并不是说,有些东西不能避免,但我们主要讨论建设长字符串在for循环,使用错误的数据结构,使O(n ^ 3)algorithsm。
#2
9
This is a pretty great thread, except there is one crucial thing you're missing. You said:
这是一条非常棒的线,除了有一件重要的事情是你错过的。你说:
Curiously, capacity, capacity+1, capacity+2, capacity-1 and even capacity-10 all yield exactly the same results. I would expect at least capacity-1 and capacity-10 to give worse results.
奇怪的是,容量、容量+1、容量+2、容量-1甚至容量-10都会产生完全相同的结果。我预计至少能力-1和能力-10会产生更坏的结果。
The source code jumps initial capacity the next highest power-of-two internally. That means that, for example, initial capacities of 513, 600, 700, 800, 900, 1000, and 1024 will all use the same initial capacity (1024). This doesn't invalidate the testing done by @G_H though, one should realize that this is being done before analyzing his results. And it does explain the odd behavior of some of the tests.
源代码在内部跳过了初始容量的下一个最高的2次方。这意味着,例如,513,600、700、800、900、1000和1024的初始容量都将使用相同的初始容量(1024)。这并不能使@G_H所做的测试无效,但是在分析结果之前应该意识到这一点。它确实解释了一些测试的奇怪行为。
This is the constructor right for the JDK source:
这是JDK源代码的构造函数:
/**
* Constructs an empty <tt>HashMap</tt> with the specified initial
* capacity and load factor.
*
* @param initialCapacity the initial capacity
* @param loadFactor the load factor
* @throws IllegalArgumentException if the initial capacity is negative
* or the load factor is nonpositive
*/
public HashMap(int initialCapacity, float loadFactor) {
if (initialCapacity < 0)
throw new IllegalArgumentException("Illegal initial capacity: " +
initialCapacity);
if (initialCapacity > MAXIMUM_CAPACITY)
initialCapacity = MAXIMUM_CAPACITY;
if (loadFactor <= 0 || Float.isNaN(loadFactor))
throw new IllegalArgumentException("Illegal load factor: " +
loadFactor);
// Find a power of 2 >= initialCapacity
int capacity = 1;
while (capacity < initialCapacity)
capacity <<= 1;
this.loadFactor = loadFactor;
threshold = (int)(capacity * loadFactor);
table = new Entry[capacity];
init();
}
#3
2
Just go with 101
. I'm not actually sure that it's needed, but it couldn't possibly be worth the effort to ever bother finding out for sure.
101就好。实际上我不确定是否需要它,但它不可能值得花精力去确定。
...just add the 1
.
…就加1。
EDIT: Some justification for my answer.
编辑:我的回答有理由。
First, I'm assuming that your HashMap
will not grow beyond 100
; if it does, you should leave the load-factor as it is. Similarly, if your concern is performance, leave the load-factor as is. If your concern is memory, you can save some by setting the static size. This might maybe be worth doing if you're cramming a lot of stuff in memory; i.e., are storing many maps, or creating heap-space-stressing-sized maps.
首先,我假设您的HashMap不会超过100;如果是这样,您应该将负载因素保留原样。类似地,如果您关心的是性能,那么请保持负载因数不变。如果您关心的是内存,可以通过设置静态大小来保存。如果你在记忆中塞满了很多东西,这可能是值得做的;即。,存储许多地图,或创建堆空间大小的地图。
Second, I choose the value 101
because it offers better readability... if I'm looking at your code afterwards and see that you've set the initial capacity to 100
and you're loading it with 100
elements, I'm going to have to read through the Javadoc to make sure that it won't resize when it reaches precisely 100
. Of course, I won't find the answer there, so I'll have to look at the source. This is not worth it... just leave it 101
and everyone is happy and no-one is looking though the source-code of java.util.HashMap
. Hoorah.
其次,我选择了value 101,因为它提供了更好的可读性……如果我在之后查看您的代码并看到您已经将初始容量设置为100,并且您正在装载100个元素,那么我将必须通读Javadoc,以确保它在达到100时不会调整大小。当然,我不会在这里找到答案,所以我得看一下来源。这不值得……只要把它放在101,每个人都很高兴,没有人会去查看java.util.HashMap的源代码。Hoorah。
Third, the claim that setting the HashMap
to the exact capacity of what you expect with a load factor of 1
"will kill your lookup and insertion performance" is just not true, even if it's made in bold.
第三,将HashMap设置为您所期望的负载因数为1的确切容量“将破坏查找和插入性能”的说法是不正确的,即使它是加粗的。
...if you have n
buckets, and you randomly assign n
items into n
buckets, yep, you're going to end up with items in the same bucket, sure... but that's not the end of the world... in practice, it's just a couple more equals comparisons. In fact, there's esp. little difference when you consider that the alternative is assigning n
items into n/0.75
buckets.
…如果你有n个桶,你随机地将n个项目分配到n个桶中,是的,你最终会得到相同桶中的项目,当然……但这还不是世界末日……在实践中,这只是更多的对等比较。事实上,当你考虑把n个项目分配到n/0.75个桶中时,差别很小。
No need to take my word for it...
没必要相信我的话……
Quick test code:
快速测试代码:
static Random r = new Random();
public static void main(String[] args){
int[] tests = {100, 1000, 10000};
int runs = 5000;
float lf_sta = 1f;
float lf_dyn = 0.75f;
for(int t:tests){
System.err.println("=======Test Put "+t+"");
HashMap<Integer,Integer> map = new HashMap<Integer,Integer>();
long norm_put = testInserts(map, t, runs);
System.err.print("Norm put:"+norm_put+" ms. ");
int cap_sta = t;
map = new HashMap<Integer,Integer>(cap_sta, lf_sta);
long sta_put = testInserts(map, t, runs);
System.err.print("Static put:"+sta_put+" ms. ");
int cap_dyn = (int)Math.ceil((float)t/lf_dyn);
map = new HashMap<Integer,Integer>(cap_dyn, lf_dyn);
long dyn_put = testInserts(map, t, runs);
System.err.println("Dynamic put:"+dyn_put+" ms. ");
}
for(int t:tests){
System.err.println("=======Test Get (hits) "+t+"");
HashMap<Integer,Integer> map = new HashMap<Integer,Integer>();
fill(map, t);
long norm_get_hits = testGetHits(map, t, runs);
System.err.print("Norm get (hits):"+norm_get_hits+" ms. ");
int cap_sta = t;
map = new HashMap<Integer,Integer>(cap_sta, lf_sta);
fill(map, t);
long sta_get_hits = testGetHits(map, t, runs);
System.err.print("Static get (hits):"+sta_get_hits+" ms. ");
int cap_dyn = (int)Math.ceil((float)t/lf_dyn);
map = new HashMap<Integer,Integer>(cap_dyn, lf_dyn);
fill(map, t);
long dyn_get_hits = testGetHits(map, t, runs);
System.err.println("Dynamic get (hits):"+dyn_get_hits+" ms. ");
}
for(int t:tests){
System.err.println("=======Test Get (Rand) "+t+"");
HashMap<Integer,Integer> map = new HashMap<Integer,Integer>();
fill(map, t);
long norm_get_rand = testGetRand(map, t, runs);
System.err.print("Norm get (rand):"+norm_get_rand+" ms. ");
int cap_sta = t;
map = new HashMap<Integer,Integer>(cap_sta, lf_sta);
fill(map, t);
long sta_get_rand = testGetRand(map, t, runs);
System.err.print("Static get (rand):"+sta_get_rand+" ms. ");
int cap_dyn = (int)Math.ceil((float)t/lf_dyn);
map = new HashMap<Integer,Integer>(cap_dyn, lf_dyn);
fill(map, t);
long dyn_get_rand = testGetRand(map, t, runs);
System.err.println("Dynamic get (rand):"+dyn_get_rand+" ms. ");
}
}
public static long testInserts(HashMap<Integer,Integer> map, int test, int runs){
long b4 = System.currentTimeMillis();
for(int i=0; i<runs; i++){
fill(map, test);
map.clear();
}
return System.currentTimeMillis()-b4;
}
public static void fill(HashMap<Integer,Integer> map, int test){
for(int j=0; j<test; j++){
if(map.put(r.nextInt(), j)!=null){
j--;
}
}
}
public static long testGetHits(HashMap<Integer,Integer> map, int test, int runs){
long b4 = System.currentTimeMillis();
ArrayList<Integer> keys = new ArrayList<Integer>();
keys.addAll(map.keySet());
for(int i=0; i<runs; i++){
for(int j=0; j<test; j++){
keys.get(r.nextInt(keys.size()));
}
}
return System.currentTimeMillis()-b4;
}
public static long testGetRand(HashMap<Integer,Integer> map, int test, int runs){
long b4 = System.currentTimeMillis();
for(int i=0; i<runs; i++){
for(int j=0; j<test; j++){
map.get(r.nextInt());
}
}
return System.currentTimeMillis()-b4;
}
Test Results:
测试结果:
=======Test Put 100
Norm put:78 ms. Static put:78 ms. Dynamic put:62 ms.
=======Test Put 1000
Norm put:764 ms. Static put:763 ms. Dynamic put:748 ms.
=======Test Put 10000
Norm put:12921 ms. Static put:12889 ms. Dynamic put:12873 ms.
=======Test Get (hits) 100
Norm get (hits):47 ms. Static get (hits):31 ms. Dynamic get (hits):32 ms.
=======Test Get (hits) 1000
Norm get (hits):327 ms. Static get (hits):328 ms. Dynamic get (hits):343 ms.
=======Test Get (hits) 10000
Norm get (hits):3304 ms. Static get (hits):3366 ms. Dynamic get (hits):3413 ms.
=======Test Get (Rand) 100
Norm get (rand):63 ms. Static get (rand):46 ms. Dynamic get (rand):47 ms.
=======Test Get (Rand) 1000
Norm get (rand):483 ms. Static get (rand):499 ms. Dynamic get (rand):483 ms.
=======Test Get (Rand) 10000
Norm get (rand):5190 ms. Static get (rand):5362 ms. Dynamic get (rand):5236 ms.
re: ↑ — there's about this →||← much difference between the different settings.
re:↑——有这个→| |←区别不同的设置。
With respect to my original answer (the bit above the first horizontal line), it was deliberately glib because in most cases, this type of micro-optimising is not good.
对于我最初的答案(在第一个水平线上面的位),它是故意的,因为在大多数情况下,这种类型的微优化是不好的。
#4
1
From the HashMap
JavaDoc:
从HashMap JavaDoc:
As a general rule, the default load factor (.75) offers a good tradeoff between time and space costs. Higher values decrease the space overhead but increase the lookup cost (reflected in most of the operations of the HashMap class, including get and put). The expected number of entries in the map and its load factor should be taken into account when setting its initial capacity, so as to minimize the number of rehash operations. If the initial capacity is greater than the maximum number of entries divided by the load factor, no rehash operations will ever occur.
一般来说,默认负载因数(.75)在时间和空间成本之间提供了很好的权衡。更高的值减少了空间开销,但增加了查找成本(反映在HashMap类的大多数操作中,包括get和put)。在设置映射的初始容量时,应该考虑映射中预期的条目数量及其负载因数,以最小化rehash操作的数量。如果初始容量大于按负载因素划分的最大条目数,则不会发生重新散列操作。
So if you're expecting 100 entries, perhaps a load factor of 0.75 and an initial capacity of ceiling(100/0.75) would be best. That comes down to 134.
因此,如果您预期有100个条目,那么负载因数为0.75和初始容量上限(100/0.75)将是最好的。下降到134。
I have to admit, I'm not certain why lookup cost would be greater for a higher load factor. Just because the HashMap is more "crowded" doesn't mean that more objects will be placed in the same bucket, right? That only depends on their hash code, if I'm not mistaken. So assuming a decent hash code spread, shouldn't most cases still be O(1) regardless of load factor?
我不得不承认,我不确定为什么查找的成本会更高。仅仅因为HashMap更“拥挤”并不意味着将有更多的对象放在同一个bucket中,对吗?这只取决于他们的哈希代码,如果我没弄错的话。假设哈希代码分布良好,那么不管负载因素如何,大多数情况下不应该仍然是O(1)吗?
EDIT: I should read more before posting... Of course the hash code cannot directly map to some internal index. It must be reduced to a value that fits the current capacity. Meaning that the greater your initial capacity, the smaller you can expect the number of hash collisions to be. Choosing an initial capacity exactly the size (or +1) of your object set with a load factor of 1 will indeed make sure that your map is never resized. However, it will kill your lookup and insertion performance. A resize is still relatively quick and would only occur maybe once, while lookups are done on pretty much any relevant work with the map. As a result, optimizing for quick lookups is what you really want here. You can combine that with never having to resize by doing as the JavaDoc says: take your required capacity, divide by an optimal load factor (e.g. 0.75) and use that as the initial capacity, with that load factor. Add 1 to make sure rounding doesn't get you.
编辑:我应该在发布之前多读一些……当然,哈希代码不能直接映射到某个内部索引。它必须被降低到适合当前容量的值。也就是说,初始容量越大,哈希冲突的数量就越小。选择一个初始容量(或+1)对象集的大小(负载系数为1)确实可以确保映射不会被调整大小。但是,它会破坏查找和插入性能。调整大小仍然相对较快,可能只会发生一次,而查找几乎都是在与映射相关的工作上进行的。因此,优化快速查找是您在这里真正需要的。您可以将其与不需要调整大小结合起来,如JavaDoc所言:取所需的容量,除以最优负载因子(例如0.75),然后使用该负载因子作为初始容量。加1以确保四舍五入没有得到。