It seems that there has been a recent rising interest in STM (software transactional memory) frameworks and language extensions. Clojure in particular has an excellent implementation which uses MVCC (multi-version concurrency control) rather than a rolling commit log. GHC Haskell also has an extremely elegant STM monad which also allows transaction composition. Finally, so as to toot my own horn just a bit, I've recently implemented an STM framework for Scala which statically enforces reference restrictions.
似乎最近对STM(软件事务内存)框架和语言扩展的兴趣日益增加。特别是Clojure有一个很好的实现,它使用MVCC(多版本并发控制)而不是滚动提交日志。 GHC Haskell还有一个非常优雅的STM monad,它也允许交易组成。最后,为了给自己的号角做一点点,我最近为Scala实现了一个静态强制引用限制的STM框架。
All of these are interesting experiments, but they seem to be confined to that sphere alone (experimentation). So my question is: have any of you seen or used STM in the real world? If so, why? What sort of benefits did it bring? What about performance? (there seems to be a great deal of conflicting information on this point) Would you use STM again or would you prefer to use some other concurrency abstraction like actors?
所有这些都是有趣的实验,但它们似乎仅限于那个领域(实验)。所以我的问题是:你们有没有在现实世界中看到过或使用过STM?如果是这样,为什么?它带来了什么样的好处?性能怎么样? (在这一点上似乎存在大量相互矛盾的信息)你会再次使用STM还是更喜欢使用像actor这样的其他并发抽象?
6 个解决方案
#1
29
I participated in the hobbyist development of the BitTorrent client in Haskell (named conjure). It uses STM quite heavily to coordinate different threads (1 per peer + 1 for storage management + 1 for overall management).
我参与了Haskell中BitTorrent客户端的爱好者开发(名为conjure)。它使用STM来协调不同的线程(每个对等1个用于存储管理,1个用于整体管理)。
Benefits: less locks, readable code.
好处:减少锁,可读代码。
Speed was not an issue, at least not due to STM usage.
速度不是问题,至少不是由于STM的使用。
Hope this helps
希望这可以帮助
#2
26
The article "Software Transactional Memory: why is it only a research toy?" fails to look at the Haskell implementation, which is a really big omission. The problem for STM, as the article points out, is that implementations must chose between either making all variable accesses transactional unless the compiler can prove them safe (which kills performance) or letting the programmer indicate which ones are to be transactional (which kills simplicity and reliability). However the Haskell implementation uses the purity of Haskell to avoid the need to make most variable uses transactional, while the type system provides a simple model together with effective enforcement for the transactional mutation operations. Thus a Haskell program can use STM for those variables that are truly shared between threads whilst guaranteeing that non-transactional memory use is kept safe.
文章“软件交易记忆:为什么它只是一个研究玩具?”没有看到Haskell实现,这是一个非常大的遗漏。正如文章所指出的,STM的问题在于,实现必须在所有变量访问事务之间进行选择,除非编译器能够证明它们是安全的(这会导致性能下降)或者让程序员指出哪些是事务性的(这会使事务变得简单)和可靠性)。然而,Haskell实现使用Haskell的纯度来避免需要使大多数变量使用事务,而类型系统提供简单模型以及事务变异操作的有效实施。因此,Haskell程序可以将STM用于线程之间真正共享的变量,同时保证非事务性内存使用保持安全。
#3
26
We use it pretty routinely for high concurrency apps at Galois (in Haskell). It works, its used widely in the Haskell world, and it doesn't deadlock (though of course you can have too much contention). Sometimes we rewrite things to use MVars, if we've got the design right -- as they're faster.
我们常常将它用于Galois(在Haskell中)的高并发应用程序。它有效,它在Haskell世界中广泛使用,并且它不会死锁(当然你可能会有太多的争用)。有时我们会重写使用MVars的东西,如果我们有正确的设计 - 因为它们更快。
Just use it. It's no big deal. As far as I'm concerned, STM in Haskell is "solved". There's no further work to do. So we use it.
只是使用它。这没什么大不了的。就我而言,Haskell中的STM已经“解决”了。没有进一步的工作要做。所以我们使用它。
#4
11
We, factis research GmbH, are using Haskell STM with GHC in production. Our server receives a stream of messages about new and modified "objects" from a clincal "data server", it transforms this event stream on the fly (by generating new objects, modifying objects, aggregating things, etc) and calculates which of these new objects should be synchronized to connected iPads. It also receives form inputs from iPads which are processed, merged with the "main stream" and also synchronized to the other iPads. We're using STM for all channels and mutable data structures that need to be shared between threads. Threads are very lightweight in Haskell so we can have lots of them without impacting performance (at the moment 5 per iPad connection). Building a large application is always a challenge and there were many lessons to be learned but we never had any problems with STM. It always worked as you'd naively expect. We had to do some serious performance tuning but STM was never a problem. (80% of the time we were trying to reduce short-lived allocations and overall memory usage.)
我们,factis research GmbH,正在使用Haskell STM和GHC进行生产。我们的服务器从临床“数据服务器”接收有关新的和修改过的“对象”的消息流,它动态地转换此事件流(通过生成新对象,修改对象,聚合事物等)并计算这些新事件中的哪一个对象应与已连接的iPad同步。它还接收来自iPad的表单输入,这些输入被处理,与“主流”合并并且还与其他iPad同步。我们将STM用于需要在线程之间共享的所有通道和可变数据结构。 Haskell中的线程非常轻量级,因此我们可以在不影响性能的情况下使用很多线程(目前每个iPad连接有5个)。构建大型应用程序始终是一项挑战,需要学习许多课程,但我们从未遇到任何STM问题。它总是像你天真的期待一样工作。我们不得不做一些严肃的性能调整,但STM从来都不是问题。 (80%的时间我们都在尝试减少短期分配和总内存使用量。)
STM is one area where Haskell and the GHC runtime really shines. It's not just an experiment and not for toy programs only.
STM是Haskell和GHC运行时真正发挥作用的一个领域。这不仅仅是一个实验,不仅仅适用于玩具程序。
We're building a different component of our clincal system in Scala and have been using Actors so far, but we're really missing STM. If anybody has experience of what it's like to use one of the Scala STM implementations in production I'd love to hear from you. :-)
我们在Scala中构建了一个不同的clincal系统组件,到目前为止一直在使用Actors,但我们确实缺少STM。如果有人体验过在生产中使用Scala STM实现之一的感受,我很乐意听取您的意见。 :-)
#5
4
We have implemented our entire system (in-memory database and runtime) on top of our own STM implementation in C. Prior to this, we had some log and lock based mechanism to deal with concurrency, but this was a pain to maintain. We are very happy with STM since we can treat every operation the same way. Almost all locks could be removed. We use STM now for almost anything at any size, we even have a memory manager implement on top.
我们在C中使用自己的STM实现实现了整个系统(内存数据库和运行时)。在此之前,我们有一些基于日志和锁的机制来处理并发,但这很难维护。我们对STM非常满意,因为我们可以以同样的方式对待每个操作。几乎所有的锁都可以被移除。我们现在几乎可以使用任何规模的STM,我们甚至还有一个内存管理器。
The performance is fine but to speed things up we now developed a custom operating system in collaboration with ETH Zurich. The system natively supports transactional memory.
性能很好,但为了加快速度,我们现在与苏黎世联邦理工学院合作开发了一个定制操作系统。系统本身支持事务内存。
But there are some challenges caused by STM as well. Especially with larger transactions and hotspots that cause unnecessary transaction conflicts. If for example two transactions put an item into a linked list, an unnecessary conflict will occur that could have been avoided using a lock free data structure.
但是STM也会带来一些挑战。特别是对于导致不必要的事务冲突的较大事务和热点。例如,如果两个事务将项目放入链接列表中,则会发生不必要的冲突,这可能是使用无锁数据结构避免的。
#6
1
I'm currently using Akka in some PGAS systems research. Akka is a Scala library for developing scalable concurrent systems using Actors, STM, and built-in fault tolerance capabilities modeled after Erlang's "Let It Fail/Crash/Crater/ROFL" philosophy. Akka's STM implementation is supposedly built around a Scala port of Clojure's STM implementation. An overview of Akka's STM module can be found here.
我目前在一些PGAS系统研究中使用Akka。 Akka是一个Scala库,用于开发使用Actors,STM和内置容错功能的可扩展并发系统,这些功能以Erlang的“Let It Fail / Crash / Crater / ROFL”理念为蓝本。据推测,Akka的STM实现是围绕Clojure的STM实现的Scala端口构建的。可以在此处找到Akka的STM模块的概述。
#1
29
I participated in the hobbyist development of the BitTorrent client in Haskell (named conjure). It uses STM quite heavily to coordinate different threads (1 per peer + 1 for storage management + 1 for overall management).
我参与了Haskell中BitTorrent客户端的爱好者开发(名为conjure)。它使用STM来协调不同的线程(每个对等1个用于存储管理,1个用于整体管理)。
Benefits: less locks, readable code.
好处:减少锁,可读代码。
Speed was not an issue, at least not due to STM usage.
速度不是问题,至少不是由于STM的使用。
Hope this helps
希望这可以帮助
#2
26
The article "Software Transactional Memory: why is it only a research toy?" fails to look at the Haskell implementation, which is a really big omission. The problem for STM, as the article points out, is that implementations must chose between either making all variable accesses transactional unless the compiler can prove them safe (which kills performance) or letting the programmer indicate which ones are to be transactional (which kills simplicity and reliability). However the Haskell implementation uses the purity of Haskell to avoid the need to make most variable uses transactional, while the type system provides a simple model together with effective enforcement for the transactional mutation operations. Thus a Haskell program can use STM for those variables that are truly shared between threads whilst guaranteeing that non-transactional memory use is kept safe.
文章“软件交易记忆:为什么它只是一个研究玩具?”没有看到Haskell实现,这是一个非常大的遗漏。正如文章所指出的,STM的问题在于,实现必须在所有变量访问事务之间进行选择,除非编译器能够证明它们是安全的(这会导致性能下降)或者让程序员指出哪些是事务性的(这会使事务变得简单)和可靠性)。然而,Haskell实现使用Haskell的纯度来避免需要使大多数变量使用事务,而类型系统提供简单模型以及事务变异操作的有效实施。因此,Haskell程序可以将STM用于线程之间真正共享的变量,同时保证非事务性内存使用保持安全。
#3
26
We use it pretty routinely for high concurrency apps at Galois (in Haskell). It works, its used widely in the Haskell world, and it doesn't deadlock (though of course you can have too much contention). Sometimes we rewrite things to use MVars, if we've got the design right -- as they're faster.
我们常常将它用于Galois(在Haskell中)的高并发应用程序。它有效,它在Haskell世界中广泛使用,并且它不会死锁(当然你可能会有太多的争用)。有时我们会重写使用MVars的东西,如果我们有正确的设计 - 因为它们更快。
Just use it. It's no big deal. As far as I'm concerned, STM in Haskell is "solved". There's no further work to do. So we use it.
只是使用它。这没什么大不了的。就我而言,Haskell中的STM已经“解决”了。没有进一步的工作要做。所以我们使用它。
#4
11
We, factis research GmbH, are using Haskell STM with GHC in production. Our server receives a stream of messages about new and modified "objects" from a clincal "data server", it transforms this event stream on the fly (by generating new objects, modifying objects, aggregating things, etc) and calculates which of these new objects should be synchronized to connected iPads. It also receives form inputs from iPads which are processed, merged with the "main stream" and also synchronized to the other iPads. We're using STM for all channels and mutable data structures that need to be shared between threads. Threads are very lightweight in Haskell so we can have lots of them without impacting performance (at the moment 5 per iPad connection). Building a large application is always a challenge and there were many lessons to be learned but we never had any problems with STM. It always worked as you'd naively expect. We had to do some serious performance tuning but STM was never a problem. (80% of the time we were trying to reduce short-lived allocations and overall memory usage.)
我们,factis research GmbH,正在使用Haskell STM和GHC进行生产。我们的服务器从临床“数据服务器”接收有关新的和修改过的“对象”的消息流,它动态地转换此事件流(通过生成新对象,修改对象,聚合事物等)并计算这些新事件中的哪一个对象应与已连接的iPad同步。它还接收来自iPad的表单输入,这些输入被处理,与“主流”合并并且还与其他iPad同步。我们将STM用于需要在线程之间共享的所有通道和可变数据结构。 Haskell中的线程非常轻量级,因此我们可以在不影响性能的情况下使用很多线程(目前每个iPad连接有5个)。构建大型应用程序始终是一项挑战,需要学习许多课程,但我们从未遇到任何STM问题。它总是像你天真的期待一样工作。我们不得不做一些严肃的性能调整,但STM从来都不是问题。 (80%的时间我们都在尝试减少短期分配和总内存使用量。)
STM is one area where Haskell and the GHC runtime really shines. It's not just an experiment and not for toy programs only.
STM是Haskell和GHC运行时真正发挥作用的一个领域。这不仅仅是一个实验,不仅仅适用于玩具程序。
We're building a different component of our clincal system in Scala and have been using Actors so far, but we're really missing STM. If anybody has experience of what it's like to use one of the Scala STM implementations in production I'd love to hear from you. :-)
我们在Scala中构建了一个不同的clincal系统组件,到目前为止一直在使用Actors,但我们确实缺少STM。如果有人体验过在生产中使用Scala STM实现之一的感受,我很乐意听取您的意见。 :-)
#5
4
We have implemented our entire system (in-memory database and runtime) on top of our own STM implementation in C. Prior to this, we had some log and lock based mechanism to deal with concurrency, but this was a pain to maintain. We are very happy with STM since we can treat every operation the same way. Almost all locks could be removed. We use STM now for almost anything at any size, we even have a memory manager implement on top.
我们在C中使用自己的STM实现实现了整个系统(内存数据库和运行时)。在此之前,我们有一些基于日志和锁的机制来处理并发,但这很难维护。我们对STM非常满意,因为我们可以以同样的方式对待每个操作。几乎所有的锁都可以被移除。我们现在几乎可以使用任何规模的STM,我们甚至还有一个内存管理器。
The performance is fine but to speed things up we now developed a custom operating system in collaboration with ETH Zurich. The system natively supports transactional memory.
性能很好,但为了加快速度,我们现在与苏黎世联邦理工学院合作开发了一个定制操作系统。系统本身支持事务内存。
But there are some challenges caused by STM as well. Especially with larger transactions and hotspots that cause unnecessary transaction conflicts. If for example two transactions put an item into a linked list, an unnecessary conflict will occur that could have been avoided using a lock free data structure.
但是STM也会带来一些挑战。特别是对于导致不必要的事务冲突的较大事务和热点。例如,如果两个事务将项目放入链接列表中,则会发生不必要的冲突,这可能是使用无锁数据结构避免的。
#6
1
I'm currently using Akka in some PGAS systems research. Akka is a Scala library for developing scalable concurrent systems using Actors, STM, and built-in fault tolerance capabilities modeled after Erlang's "Let It Fail/Crash/Crater/ROFL" philosophy. Akka's STM implementation is supposedly built around a Scala port of Clojure's STM implementation. An overview of Akka's STM module can be found here.
我目前在一些PGAS系统研究中使用Akka。 Akka是一个Scala库,用于开发使用Actors,STM和内置容错功能的可扩展并发系统,这些功能以Erlang的“Let It Fail / Crash / Crater / ROFL”理念为蓝本。据推测,Akka的STM实现是围绕Clojure的STM实现的Scala端口构建的。可以在此处找到Akka的STM模块的概述。